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EXECUTIVE  SUMMARY 


PROBLEM 

The  Navy  needs  to  reduce  training  costs  while  at  the  same 
time  train  faster,  to  a  higher  level,  and  save  lives  and  equipment. 
These  improvements  require  that  we  prioritize  training  technologies 
for  Research  and  Development  (R&D) ,  and  select  the  technologies 
with  the  greatest  potential  for  return  on  the  R&D  investments. 

OBJECTIVE 

This  report  documents  an  effort  to  identify  training 
technology  priorities  to  guide  R&D  programs  at  the  Naval  Air 
Warfare  Center  Training  Systems  Division  (NAWCTSD;  formerly  the 
Naval  Training  Systems  Center;  NAVTRASYSCEN) ,  with  potential 
generalizability  across  the  military. 

METHOD 

A  Simulation  and  Training  Technology  Survey  (STTS)  was 
developed  and  administered  to  training  experts  in  government  and 
private  industry.  Results  from  the  survey  were  examined  for  R&D 
implications. 

RESULTS  AND  CONCLUSIONS 

This  STTS  identifies  and  defines  91  training  technologies  that 
are  suitable  topics  for  R&D  programs.  The  major  finding  from  the 
survey  is  that  training  experts  consider  most,  if  not  all,  of  these 
technologies  important  topics  for  future  R&D.  A  review  of  related 
efforts  to  define  training  technology  priorities  for  R&D  showed  a 
high  degree  of  similarity  with  the  current  results. 

Relative  priorities  for  specific  technologies  are  identified, 
with  special  consideration  given  to  validity  issues. 

RECOMMENDATIONS 

o  Consider  all  technology  areas  identified  in  the  STTS  for 
development  and  implementation. 

o  Prioritize  selected  topics  for  R&D  based  on  findings 
presented  in  this  report  together  with  the  benefits,  costs,  risks, 
and  applicability  of  particular  technical  developments.  Criteria 
such  as  described  in  the  NAVPERSDEVCEN-NAVTRASYSCEN  Plan  (Appendix 
B)  should  help  guide  this  process. 

o  Improve  Navy  research  and  test  facilities  to  better 

evaluate  promising  technologies. 
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o  Improve  evaluations  of  fielded  training  systems  to  assure 
their  proper  use  and  cost  effectiveness. 
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INTRODUCTION 


PROBLEM 

"People  and  associated  training  requirements  cost  60%  of 
weapon  systems  life  cycle  costs"  (Investment  strategy,  1992,  p.7). 
With  ever  shrinking  resources,  the  Navy  needs  to  reduce  these  costs 
while  at  the  same  time  train  faster,  to  a  higher  level,  and  save 
lives  and  equipment.  These  improvements  require  better  and  less 
expensive  training  technology.  In  turn,  this  requires  that  we 
prioritize  technologies  for  Research  and  Development  (R&D) ,  and 
select  the  technologies  with  the  greatest  potential  for  return  on 
the  R&D  investments. 

OBJECTIVE 

Technologies  are  prioritized,  implicitly  or  explicitly,  each 
time  a  training  R&D  program  is  initiated.  Occasionally,  efforts 
are  made  to  define  shortcomings  in  training  technologies  more 
globally,  to  guide  R&D  across  a  range  of  programs.  This  report 
documents  an  effort  to  identify  R&D  priorities  to  guide  programs  at 
the  NAWCTSD  (formerly  NAVTRASYSCEN) ,  with  potential 
generalizability  across  the  military.  This  effort  took  the  form  of 
a  Simulation  and  Training  Technology  Survey  (STTS) . 

BACKGROUND 

Additional  surveys  and  analyses  that  can  help  identify 
training  technologies  for  future  R&D  efforts  include;  (a)  Training 
Systems  Technology  Assessment  Plan  of  Action  and  Milestones  (1991)  ; 
(b)  NAVTRASYSCEN  Inputs  to  Navy  Training  Appraisal  (1989);  (c)  1991 
Navy  Training  Appraisal  Space  and  Electronic  Warfare  Issue 
Development  (1991) ;  (d)  Desert  Shield  and  Desert  Storm  Implications 
for  Future  U.S.  Force  Requirements  (Collins,  1991);  (e)  Navy 
Training  Plans;  (f)  Report  of  Industry  Task  Force  Navy  Training- 
2000  (1990) ;  (g)  Investment  Strategy  for  Training  Systems 

Technology  Area  Exploratory  Development  (NAVPERSDEVCEN-NAVTRASYSCEN 
Plan,  1992). 

The  latter  two  reports,  most  closely  related  to  the  current 
effort,  are  discussed  briefly  in  the  following. 

Training-2000 

In  February  1989,  Rear  Admiral  Cressy,  Director  of  Aviation 
Manpower  and  Training  with  the  Chief  of  Naval  Training,  initiated 
an  effort  to  identify  training  technologies  that  can  have 
application  to  naval  training,  particularly  aviation,  in  the  year 
2000  and  beyond.  Toward  this  end,  a  committee  of  18  individuals 
from  government  and  industry  identified  18  technologies  that  are 
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ready  for  exploitation  through  R&D  (Appendix  A) .  Fifteen  of  these 
18  technologies  also  are  identified  (conceptually)  as  important  R&D 
topics  in  the  STTS  (presented  later  in  this  report)  .  The  three 
exceptions  are:  Fiber  Optics,  Telecommunication  Satellites,  and 
Training  Effectiveness  Assessment  Tools. 

The  committee  also  recommended  funding  for  research  on  six 
research  issues  that  encompass  nine  training  technologies  (Appendix 
A)  .  Again,  there  is  considerable  overlap  between  this  list  and 
technologies  identified  for  R&D  by  the  STTS.  Except  for  Mission 
Planning  and  Mission  Rehearsal,  all  technology  areas  recommended 
for  funding  in  the  Training-2000  report  also  are  recommended  for 
R&D  in  the  STTS.  The  STTS  includes  Mission  Planning  and  Mission 
Rehearsal  as  Military  Training  Requirements,  i.e.,  operational 
objectives  toward  which  the  technologies  are  directed. 

The  close  similarity  in  the  technologies  identified  in  these 
two  efforts  shows  high  consensu?  in  the  military  training  community 
concerning  the  most  critical  technologies  to  advance  through  R&D. 

NAVPERSDEVCEN-NAVTRASYSCEN  PLAN 

The  NAVPERSDEVCEN-NAVTRASYSCEN  Plan  (1992),  developed  by  the 
Naval  Personnel  Research  and  Development  Center  (NAVPERSDEVCEN) 
with  contributions  from  the  NAVTRASYSCEN,  gives  plans  and  rationale 
for  Exploratory  Development  on  military  training  issues.  The 
rationale  consists  of  13  trends  in  Navy  training  (e.g.,  fewer 
people)  and  22  requirements  for  dealing  with  the  trends  (e.g., 
widely  available  cross  training) .  The  plan  recommends  R&D  on  17 
technologies  (e.g.,  networks/distributed  training)  to  satisfy  the 
requirements. 

All  17  technologies  also  are  recommended  for  R&D  by  the  STTS 
effort.  This  further  supports  the  priority  of  the  technologies 
identified. 

In  addition  to  the  above,  the  Plan  also  presents  a  more 
extensive  list  of  technologies  with  associated  definitions.  This 
list  contains  16  technologies  categorized  as  delivery  technologies, 
20  as  instructional  processes  and  methods,  and  32  as  supporting 
technologies.  Together,  the  two  lists  show  specific  technologies 
that  support  more  general  technologies  and  how  the  technologies 
support  Navy  requirements.  This  framework  is  similar  to  the 
structure  of  the  STTS.  Major  conclusions  from  the  Plan  are  shown 
in  Appendix  B. 

The  technologies  identified  in  the  Plan  are  important  areas 
for  R&D  and  operational  implementation. 
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METHOD 


The  STTS  (Appendix  C)  was  developed  by  research  personnel  at 
the  NAVTRASYSCEN  with  significant  contributions  from  private 
industry.  Technologies  were  initially  defined  in  brief  papers, 
written  in  support  of  the  survey  by  technical  experts  at  the 
NAVTRASYSCEN.  The  papers  covered  the  major  R&D  thrust  areas 
currently  underway  or  in  plans  at  the  NAVTRASYSCEN.  Technologies 
identified  in  the  papers  became  the  content  of  the  survey.  The 
papers  were  appended  to  the  survey  to  serve  as  background 
information  for  completing  the  survey.  Draft  copies  of  the  survey 
were  distributed  to  individuals  at  the  NAVTRASYSCEN  and  in  private 
industry  for  review  and  critique.  The  survey  was  modified  based  on 
comments  from  the  reviewers. 

The  STTS  has  a  hierarchical  structure  to  relate  technologies 
at  different  levels  of  specificity  and  to  show  how  each  technology 
supports  military  objectives.  At  the  top  of  the  hierarchy,  11 
Military  Training  Requirements  illustrate  broad  operational 
objectives  toward  which  R&D  programs  must  be  directed.  The  second 
level  contains  nine  Delivery  Methods/ Strategies,  which  are 
approaches  for  administering  instruction  to  satisfy  the  military 
requirements.  At  the  third  level,  17  Training  Technologies  define 
general  capabilities  that  allow  the  Delivery  Methods /Strategies  to 
be  implemented.  Sixty  five  Enabling  Technologies  are  included  at 
the  fourth  and  final  level,  to  identify  specific  capabilities  that 
support  the  development  of  Training  Technologies.  The  Enabling 
Technologies  have  development  milestones. 

For  example,  one  of  the  11  Military  Training  Requirements 
listed  in  the  survey  is,  Save  training  resources.  Embedded 
Training  is  one  of  nine  Delivery  Methods/ Strategies  proposed  to 
satisfy  this  and  other  military  training  requirements.  Training 
and  Organizational  Management  is  identified  as  one  of  three 
Training  Technology  areas  that  require  development  in  order  to 
implement  Embedded  Training.  Training  and  Organizational 
Management  is  proposed  to  depend  on  Enhanced  lesson  and  scenario 
authoring  (along  with  two  other  Enabling  Technologies) .  Finally, 
the  survey  predicts  that  a  prototype  tool  for  automated  scenario 
authoring  will  be  available  by  1995,  and  the  tool  will  be  ready  for 
general  use  by  the  year  2000.  This  indicates  that,  given  the 
current  pace  of  R&D,  lesson  and  scenario  authoring  technologies 
should  be  ready  to  support  Embedded  Training  by  2000.  It  also 
predicts  that  Embedded  Training  will  more  effectively  satisfy 
operational  requirements  when  this  and  the  other  supporting 
technologies  identified  in  the  STTS  are  developed. 
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The  respondents'  main  objective  was  to  identify  the  importance 
of  advancing  technologies  identified  at  each  level  in  the 
hierarchy,  relative  to  competing  methods  and  technologies  of  which 
they  were  aware.  This  was  accomplished  by  rating  each  item  on  a 
five-point  scale,  ranging  from  (1)  =  Very  Critical  to  (5)  = 
Minimally  Critical. 

The  respondents  also  rated  their  level  of  competence  for  each 
criticality  rating  given.  Competence  ratings  were  made  on  a  five- 
point  scale,  where  (1)  =  Very  Knowledgeable  and  (5)  =  Limited 
Knowledge.  The  survey  requested  information  on  the  rater's 
background /experience,  and  invited  the  respondent  to  comment  on  any 
aspect  of  the  survey  and  to  add  and  rate  new  requirements,  methods, 
and  technologies,  as  appropriate. 

SURVEY  IMPLEMENTATION 

In  Spring  1992,  the  final  version  of  the  survey  was  delivered 
to  18  individuals  from  the  NAVTRASYSCEN  and  private  industry  who 
agreed  to  complete  the  survey.  The  respondents  were  solicited  from 
among  those  who  contributed  to  the  development  of  the  survey  and 
from  the  membership  of  the  National  Security  Industry  Association 
R&D  Subcommittee. 

Thirteen  of  the  18  people  responding  to  the  solicitation 
indicated  an  educational  background  in  engineering/physical 
science;  three  identified  behavioral  science  as  their  major 
educational  qualification;  one  was  educated  at  the  U.S.  Military 
Academy;  and  educational  information  was  missing  from  one 
respondent.  Their  level  of  experience  in  training  areas  ranged 
between  six  and  35  years,  with  a  mean  of  over  20  years. 

Individuals  and  organizations  contributing  directly  to  the 
development  of  the  survey  and  providing  responses  to  the  survey  are 
listed  in  Appendix  D. 
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RESULTS  AND  DISCUSSION 

All  survey  rating  data  are  reported  as  means  and  standard 
deviations.  Inferential  tests  of  significance  were  not  made 
because  the  sample  of  raters  was  considered  too  small  in  number  and 
insufficiently  diverse  in  background  to  adeguately  assess  the 
diversity  of  technologies  included  on  the  survey,  and,  therefore, 
to  generalize  the  findings  to  a  larger  community  of  training 
experts . 

In  particular,  because  only  three  of  the  respondents  had 
significant  experience  in  the  behavioral  sciences,  and  one  of  these 
individuals  completed  only  a  portion  of  the  survey,  the  ratings 
corresponding  to  behavioral  elements  of  the  survey  probably  are 
least  representative  of  general  expert  opinion.  As  a  group,  the 
respondents  also  rated  themselves  as  less  competent  on  the 
behaviorally  oriented  items,  thus  reinforcing  the  suspicion  that 
the  judgements  given  to  behavioral  elements  are  less  representative 
of  general  expert  opinion. 

Marketing  strategies  could  place  additional  influences  on 
survey  responses  from  individuals  in  private  industry. 
Technologies  with  high  economic  value  to  private  industry  could  be 
attributed  inappropriately  high  levels  of  criticality. 

Caveats  such  as  these  should  influence  interpretations  of  the 
findings,  but  should  not  negate  the  major  contributions.  The 
current  effort,  building  upon  previous  efforts,  has  helped  define, 
reinforce  and  extend  the  rationale,  operations,  and  plans  of  the 
major  training  and  simulation  industry.  It  involved  much  effort 
from  many  experts,  and  makes  significant  contributions  toward 
improving  and  updating  commonly  shared  conceptions  of  advanced  and 
advancing  technologies.  Other  such  efforts  must  build  on  the 
current  effort  to  help  assure  efficient  progress,  through 
coordinated  conceptualizations,  toward  common  training  goals. 

DELIVERY  METHODS 

The  mean  Criticality  ratings  associated  with  the  Delivery 
Methods  ranged  between  1.8  and  3.2  (Table  1).  Seven  of  the  nine 
Methods  were  rated  at  least  "Moderately  Critical"  (3.0).  The 
overall  mean  and  standard  deviation,  across  all  Methods,  were  2.6 
and  1.0,  respectively.  This  indicates  that  the  respondents 
differed  among  themselves  about  the  criticality  of  specific 
Delivery  Methods,  but  overall  considered  the  Methods  to  be 
important  subjects  for  R&D. 

A  2.6  overall  Criticality  rating  is  considered  especially 
favorable  because  the  criterion  for  the  highest  rating  was 
extremely  stringent.  The  highest  (Very  Critical)  rating  means  that 
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Table  1 


DELIVERY  METHODS: 
CRITICALITY  &  COMPETENCE  RATINGS 
(N=18) 


DELIVERY 

METHOD 

CRITICALITY 

COMPETENCE  | 

RANK 

RATING 

STD 

DEV 

RANK 

RATING 

STD 

DEV 

Tom  Tninmg 

1.5 

1.8 

.8 

2.5 

2.7 

.8 

Deployed  Tninint 

1.5 

1.8 

.8 

1.0 

2.5 

IH 

Muhi  FUyer  Exetciie* 

3.0 

2.3 

1.2 

7.0 

2.9 

Virtual  RealiQr 

4.0 

2.5 

.8 

5.0 

2.8 

n 

Emulation 

5.5 

2.8 

.8 

5.0 

2.8 

n 

Embedded  Traming 

5.5 

2.8 

.9 

8.0 

3.0 

Interactive  Couraeware 

7.0 

3.0 

1.2 

5.0 

2.8 

n 

Waigaming  on  Rangea 
with  Operational  Equip 

8.5 

3.2 

1.3 

9.0 

3.3 

m 

Low  Coat  PC 

Applications 

8.5 

3.2 

1.3 

2.5 

2.7 

B 

OVERALL 
MEANS  AND 
STD  DEV 

2.6 

1.0 

2.8 

III^J 

Key:  1  =  Very  Critical/Very  Knowledgeable 
3  =  Moderately  Critical/Knowledgeable 
5  =  Minimally  Critical/Limited  Knowledge 


"Progress  is  more  important  than  for  any  other  capability."  This 
high  criterion  encourages  discrimination  among  the  training  areas. 
But  it  also  severely  limits  the  number  of  areas  that  can  qualify 
for  a  "Very  Critical"  rating,  which  would  decrease  the  average 
ratings  on  all  survey  items. 

Team  Training  and  Deployed  Training  tied  for  the  highest 
Criticality  rating  among  the  Delivery  Methods.  Multi-Player 
Exercises  ranked  third,  and  Virtual  Reality  ranked  fourth.  The 
high  ratings  given  to  these  four  areas  reflects  their  recent 
emphasis  in  the  training  community. 
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The  top  ratings  given  to  the  two  Delivery  Methods  used  to 
teach  people  to  work  as  collective  units  —  Team  Training  and 
Multi-player  Exercises  —  shows  strong  interest  in  "collective 
training"  issues  in  general.  Team  Training,  the  highest  rated 
Delivery  Method,  focuses  on  instructional  elements  of  collective 
training,  such  as  exercise  development,  performance  measurement, 
and  debriefing  techniques.  Multi-Player  Exercises,  rated  in  third 
place,  deals  mainly  with  engineering  aspects  of  collective 
training,  in  the  form  of  networking. 

The  remaining  five  Delivery  Methods  —  Emulation,  Embedded 
Training,  Interactive  Courseware,  Wargaming  on  Ranges,  and  Low  Cost 
PC  Applications  —  were  rated  lower  in  criticality,  in  the  order 
given. 

As  a  group,  the  respondents  considered  themselves  to  be  at 
least  Moderately  Competent  with  the  Delivery  Methods,  except  for 
Wargaming  on  Ranges  with  Operational  Equipment  (which  received  a 
3.3  rating).  The  overall  mean  for  Competence  was  2.8,  with  a 
standard  deviation  of  1.0. 

There  appears  to  be  no  significant  relationship  between  the 
overall  Competence  ratings  and  Criticality  ratings.  This  adds 
validity  to  the  results:  the  respondents  as  a  group  did  not  rate  a 
Delivery  Method  as  more  important  simply  because  they  were  more 
familiar  with  it. 

A  further  observation  is  that  the  respondents*  Criticality 
ratings  were  less  variable  for  Delivery  Methods  rated  higher  in 
Criticality.  This  is  shown  in  Table  1,  where  all  but  one  of  the 
six  Methods  rated  highest  in  Criticality  have  lower  standard 
deviations  (.8  and. 9)  than  the  three  Methods  rated  lowest  in 
Criticality  (1.2  and  1.3).  Apparently,  the  Criticality  of  the 
highest  rated  items  was  sufficiently  evident  to  create  high 
agreement  among  the  raters. 

A  separate  analysis  of  responses  for  just  the  three  behavioral 
respondents  revealed  a  somewhat  different  ordering  of  Criticality 
and  Competence,  as  shown  in  Table  2.  Because  ratings  for 
engineering  and  behavioral  items  differed  dramatically  depending  on 
the  background  of  the  rater  and  representation  of  behavioral  raters 
was  so  low,  it  seems  appropriate  to  consider  the  behavioral  ratings 
separately. 

Table  2  shows  that,  for  the  behavioral  respondents.  Team 
training  ranked  highest  and  Wargaming  on  Ranges  ranked  among  the 
lowest.  These  priorities  are  nearly  identical  to  those  found  for 
the  total  group  of  respondents.  But,  Interactive  Courseware  and 
Low  Cost  PC  Applications  ranked  high  (second  and  third)  for  the 
behavioral  respondents  and  low  (last  and  second  last)  for  the  total 
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Table  2 

DELIVERY  METHODS: 

BEHAVIORAL  CRITICALITY  &  COMPETENCE  RATINGS 


(N=3) 


CRITICALITY 

COMPETENCE 

DELIVERY 

METHOD 

RANK 

RATING 

RANGE 

RANK 

RATING 

RANGE 

Team  Trainiiig 

■B 

1.0 

1 

2.5 

1.5 

B9 

Low  Com  PC 

Applicatioaa 

2.5 

1.5 

IB 

2.5 

1.5 

B 

Interactive 

Counewaie 

2.5 

1.5 

IB 

2.5 

1.5 

B 

Embedded  Tniniog 

5.5 

2.0 

2 

2.5 

1.5 

mm 

Viftual  RealiQ' 

5.5 

2.0 

2 

6.0 

2.5 

2-3 

Deployed  Training 

5.5 

HB 

2 

m 

3.0 

3 

Emulation 

5.5 

2.0 

1-3 

5.0 

2.0 

1-3 

Waigaminc  on  Rangei  with 
Operational  Equ^ 

8.5 

BH 

3 

B 

IB 

3 

Multi  Player 

Exerciaea 

8.5 

IB 

Bi 

9.0 

3.5 

B 

OVERALL 

MEANS  & 

RANGES 

2.0 

1-4 

2.2 

1-4 

Key:  1  =  Very  Critical/Very  Knowledgeable 
3  =  Moderately  Critical/Knowledgeable 
5  =  Minimally  Critical/Liraited  Knowledge 


group.  Also,  Deployed  Training  tied  for  first  rank  in  the  overall 
group  ratings,  but  was  ranked  lower  by  behavioral  raters. 

As  expected.  Competence  ratings  for  the  behavioral  respondents 
also  showed  large  differences  compared  with  engineering 
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respondents,  mainly  with  higher  Competence  for  Interactive 
Courseware  and  lower  Competence  for  Deployed  Training. 

TRAINING  TECHNOLOGIES 

Table  3  shows  that  16  of  the  17  Training  Technologies  (94%) 
have  mean  Criticality  ratings  ranging  between  1.7  and  2.9, 
indicating  Moderately  Critical  or  higher.  The  one  exception. 
Training  and  Organizational  Management,  has  a  mean  Criticality 
rating  of  3.5.  The  overall  mean  and  standard  deviation  across  all 
Training  Technologies  were  2.6  and  1.0,  respectively.  These 
overall  measures  are  identical  with  those  obtained  for  Delivery 
Methods,  and  the  implications  are  the  same:  respondents  differed 
with  each  other  regarding  the  criticality  of  specific  Technologies, 
but  they  generally  agreed  that  the  Technologies  are  important  areas 
for  R&D. 

Visual/Sensor  Simulation  —  which  applies  to  a  variety  of 
Delivery  Methods  —  received  the  highest  Criticality  rating  among 
the  Training  Technologies.  Carrier  Based  Weapons  System  Trainer 
received  the  second  highest  rating.  Four  tied  for  third  highest 
rating:  the  Visual  Displays  component  of  Virtual  Reality, 
Networking,  Team  Performance  Measurement  Systems,  and  Team  Training 
Systems.  The  high  ratings  for  the  two  visual  technologies  included 
on  the  survey  reflects  a  long-term  community  emphasis  on  visual 
issues. 

Criticality  ratings  for  the  remaining  11  Training  Technologies 
gradually  declined,  with  only  small  (.2  or  less)  differences  among 
successively  ranked  items.  The  one  exception  is  Training  and 
Organizational  Management  —  the  lowest  rated  item,  which  was  rated 
.6  lower  in  Criticality  than  the  next  to  lowest  rated  item. 

Similar  to  Delivery  Methods,  the  Criticality  ratings  were  less 
variable  for  Technologies  rated  more  Critical.  This  is  shown  in 
Table  3,  where  all  but  one  (Networking)  of  the  six  Training 
Technologies  ranked  highest  in  Criticality  have  relatively  low 
standard  deviations  (between  .6  and  .9),  whereas  all  but  one 
(Tactile  Displays)  of  the  eleven  Training  Technologies  rated  lower 
in  Criticality  have  higher  standard  deviations  (between  1.0  and 
1.3).  Again,  the  Criticality  of  the  highest  rated  items  was 
sufficiently  evident  to  produce  high  agreement  among  the  raters. 

The  respondents  considered  themselves  to  be  moderately 
knowledgeable  on  the  Training  Technologies,  with  Expert  Systems, 
Carrier-Based  Weapon  System  Trainer  and  Visual  Displays  receiving 
che  highest  Competency  ratings  (2.3,  2.5,  and  2.5,  respectively). 
They  rated  themselves  least  competent  with  Training  and 
Organizational  Management,  Eye  Movement  Transducers  and  Team 
Training  Needs  Analysis  System  (3.8,  3.6,  and  3.5,  respectively). 
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TRAINING  TECHNOLOGIES: 
CRITICALITY  AND  COMPETENCE  RATINGS 
(N=18) 


Key:  1  =  Very  Critical/ Very  Knowledgeable 
3  =  Moderately  Critical/Knowledgeable 
5  =  Minimally  Critical/Liroited  Knowledge 


20 


Special  Report  93-005 


The  overall  mean  Competency  rating  and  standard  deviation  were  2.9 
and  1.0,  respectively,  nearly  identical  to  those  discussed  earlier 
for  Delivery  Methods. 

In  contrast  with  the  Delivery  Methods  results,  the  respondents 
rated  Training  Technologies  with  which  they  were  more  competent  as 
also  more  Critical.  Table  3  shows  that  except  for  Team  Performance 
Measurement  System,  all  of  the  Technologies  ranked  in  the  top  ten 
in  Criticality  are  also  ranked  in  the  top  ten  in  Competence. 

This  tendency  to  rate  more  familiar  topics  as  more  critical  is 
quite  evident  for  the  behaviorally  oriented  items.  The  five 
Technologies  with  clear  behavioral  orientations  —  Team  Performance 
Measurement  System,  Team  Training  Needs  Analysis  System, 
Instructor/Observer  Training  System,  Training  and  Organizational 
Management,  and  Automated  Instructor  Support  —  have  lower 
ratings  in  both  Criticality  and  Competence  compared  with  the 
remaining  12  technologies,  that  have  stronger  engineering 
orientations.  The  mean  Criticality  ratings  for  the  five 
"behavioral”  and  12  "engineering"  items  are  2.7  and  2.3, 
respectively.  The  mean  Competence  ratings  for  the  for  the  five 
behavioral  and  12  engineering  items  are  3.3  and  2.7,  respectively. 

Greater  familiarity  with  a  technology  may  create  a  perception 
of  greater  criticality.  Alternately,  a  perception  of  greater 
criticality  may  stimulate  efforts  to  become  more  familiar  with  an 
area.  It  seems  that  the  former  was  a  strong  factor,  since  the 
respondents'  predominantly  engineering  background  would  require 
familiarity  with  the  engineering  technologies,  which  allows 
relatively  little  time  to  develop  competence  with  behavioral 
technologies,  even  those  that  may  be  considered  critical.  In 
either  case,  it  would  be  more  difficult  to  accurately  assess  the 
importance  of  a  technology  with  which  one  has  little  familiarity. 
So,  criticality  ratings  should  be  viewed  with  increased  caution  for 
technologies  with  low  competence  ratings;  these  being  primarily  the 
technologies  with  strong  behavioral  orientations. 

Table  4  shows  that  the  three  highest  Criticality  ratings  and 
one  of  the  second  highest  ratings  assigned  by  the  three  behavioral 
respondents  all  are  associated  with  Team  Training.  This  is 
consistent  with  the  high  ratings  given  by  the  total  group  of 
respondents  to  Team  Training  as  a  Delivery  Method,  but  contrasts 
with  the  much  lower  ratings  given  to  these  four  Training 
Technologies  by  the  total  group.  (In  the  total  group  analysis,  two 
of  the  four  Training  Technologies  tied  with  four  others  for  third 
highest  rating  and  the  other  two  tied  for  12th  rank.)  This  shows 
agreement  between  practitioners  in  the  engineering  and  behavioral 
disciplines  about  a  training  capability  required  by  the  military, 
but  reveals  fundamental  differences  concerning  technologies 
required  to  achieve  the  capabilities. 
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Table  4 


TRAINING  TECHNOLOGIES: 

BEHAVIORAL  CRITICALITY  AND  COMPETENCE  RATINGS 

(N=3) 


TRAINING 

TECHNOLOGY 

CRITICALITY 

COMPETENCE 

RANK 

RAT 

ING 

RANGE 

RANK 

RAT 

ING 

RANGE 

Team  Tnif  Syaiem 

2.0 

n 

1-2 

3.0 

2.0 

2 

Team  Peif  Meaa  Syt 

2.0 

wm 

3.0 

2.0 

2 

Team  Tni(  Needs 

Analytts  SyMem 

BB 

B 

IB 

Bl 

B 

2 

rnsIr/Obierver 

Traming  System 

m 

B 

2 

IB 

B 

2 

Expert  Systems 

5.0 

Bl 

MBM 

3.0 

2.0 

Bl 

AutoaUd  Seen- 

ftrioOen  A  Oatl 

Bl 

2.0 

1-3 

16.0 

3.3 

1-3 

Vii/Senf  Sim 

8.0 

2.5 

2-3 

11.5 

3.0 

2-4 

Tactile  Dispb 

8.0 

2.5 

2-3 

6.5 

2.5 

2-3 

CV-BasedWST 

8.0 

2.5 

2-3 

11.5 

3.0 

3 

Networiung 

13.0 

3.0 

2-4 

6.5 

2.5 

3-4 

Thfeat  Modig  A 

Com  Data  Bases 

13.0 

B 

3 

11.5 

B 

3 

Visual  Displays 

13.0 

3.0 

3 

11.5 

3.0 

3 

EogrTeeb 

13.0 

3.0 

3 

11.5 

3.0 

3.0 

Audio  Dispis 

13.0 

3.0 

3.0 

11.5 

3.0 

3 

Auto  bistr  Sup 

13.0 

3.0 

2-4 

11.5 

3.0 

2-4 

Eye  Movement 

Traosducen 

13.0 

B 

3 

11.5 

B 

3 

Tog  A  Oigan  M^t 

17.0 

3.5 

3-4 

17.0 

3.5 

3-4 

OVERALL  MEANS  A  RANGES 

2.5 

1-4 

1-4 

Key:  1  =  Very  Critical/Very  Knowledgeable 
3  =  Moderately  Critical/Knowledgeable 
5  =  Minimally  Critical/Limited  Knowledge 
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This  discrepancy  in  ratings  between  engineering  and  behavioral 
specialists  also  was  found  for  Automated  Scenario  Generation  and 
Control  and  Expert  Systems.  These  items  were  in  the  highest  two 
ranks  for  the  behavioral  raters,  but  ranked  eighth  and  tenth, 
respectively,  for  the  total  group. 

Behavioral  respondents  were  in  exact  agreement  with  the  total 
group  concerning  the  four  Training  Technologies  ranked  lowest  in 
Criticality:  Audio  Displays,  Automated  Instructional  Support,  Eye 
Movement  Transducers,  and  Training  and  Organizational  Management. 
It  may  be  somewhat  surprising  that  Automated  Instructional  Support 
and  Training  and  Organizational  Management,  major  behavioral 
areas,  ranked  so  low  with  the  behavioral  raters.  But  this  can  be 
explained  by  the  fact  that  only  two  behavioral  responses  were 
available  for  each  of  the  two  items. 

ENABLING  TECHNOLOGIES 

Table  5  shows  mean  ratings  given  to  Enabling  Technologies, 
grouped  according  to  the  Training  Technologies  and  Delivery  Methods 
they  support.  As  was  found  for  Delivery  Methods  and  Training 
Technologies,  most  —  57  of  65  (88%)  —  of  the  Enabling 
Technologies  were  rated  Moderately  Critical  or  higher.  The  range 
of  the  Criticality  ratings  —  1.5  through  3.9  —  is  considerably 
larger  than  obtained  for  Delivery  Methods  and  Training 
Technologies,  presented  in  the  earlier  analyses.  This  is  expected, 
because  the  larger  number  of  Enabling  Technologies  increases  the 
chances  for  variation. 

The  overall  mean  Criticality  rating  for  Enabling  Technologies 
was  2.5;  the  overall  standard  deviation  was  1.0.  Again,  these 
results  are  nearly  identical  to  results  discussed  earlier  for  the 
two  higher-order  categories.  The  implications  are  also  similar  to 
the  earlier  analyses:  these  results  show  that  the  respondents 
consider  the  Enabling  Technologies  to  be  important  areas  for  R&D. 

Criticality  ratings  given  to  the  22  Enabling  Technologies 
associated  with  the  four  highest  rated  Delivery  Methods  —  Team 
Training,  Deployed  Training,  Multi-Player  Exercises,  and  Virtual 
Reality,  with  only  two  exceptions,  have  Moderately  Critical  or 
higher  ratings.  The  two  exceptions  are  Network  Security  and  Head- 
mounted  Stereo  (rated  3.1  and  3.2,  respectively.)  This  indicates 
that,  in  the  respondents'  judgements,  the  Enabling  Technologies 
were  appropriate  issues  for  investigation  to  advance  the  most 
important  Delivery  Methods. 
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TABLE  5 


ENABLING  TECHNOLOGIES;  CRITICALITY  AND  COMPETENCE  RATINGS 

(N=18) 


DELIVERY 

TRAINING 

ENABLING 

CRITICALITY 

1  COMPETENCE  | 

METHOD* 

TECH. 

TECH. 

RANK 

RATING 

STD 

RANK 

RATING 

STD 

DEV 

DEV 

1  TEAM 

Team 

-Exercise 

32.3 

2.5 

B 

31 

.9 

1  TRAINING 

Training 

devel 

■1 

1 

System 

-Team  trng 

32.5 

2.5 

35 

.7 

32.5 

2.5 

H 

18 

1.0 

Team 

-Teamwork 

20 

2.3 

.8 

35 

3.1 

Perf 

skills 

Meas  Sys 

meas 

-Trng  eval 

20 

2.3 

.9 

38.5 

3.2 

meas 

-Diagnos 

mechs 

38 

2.6 

.8 

45 

3.3 

■ 

Team 

-Needs 

42 

2.7 

45 

3.3 

.8 

Trng 

anal 

Needs 

survey 

Anal  Sys 

-Mult  anal 
of  perf 

50 

2.9 

.8 

55 

3.5 

.9 

Instr/ 

-Task/tecun 

38 

2.6 

.9 

45 

3.3 

n 

Observer 

related 

Trng  Sys 

guided 

fdbk 

H 

-Trng 

delivery 

50 

2.9 

.8 

27.5 

2.9 

■ 

-Pre-trng 

capability 

diagnosis 

56 

3.0 

.6 

55 

3.5 

■ 

*  Enabling  Technologies  were  not  identified  for  Interactive 
Courseware,  Wargaining  on  Ranges  With  Operational  Equipment, 
Emulation,  and  Low  Cost  Operations.  These  Delivery  Methods  were 
omitted  from  this  table. 

Key;  1  =  Very  Critical/Very  Knowledgeable 
3  =  Moderately  Critical/Knowledgeable 
5  =  Minimally  Critical/Limited  Knowledge 
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DELIVERY 

TRAINING 

ENABLING 

CRITICALITY 

COMPETENCE 

METHOD 

TECHNOLOGY 

TECHNOLOGY 

RANK 

RAT 

STD 

RANK 

RAT 

STD 

ING 

DEV 

ING 

DEV 

DEPLOYED 

Carrier- 

-Reconf ig 

12.5 

2.1 

1.2 

5.5 

2.3 

ID 

TRAINING 

Based 

low  maint 

mg 

Weapons 

cockpit 

HI 

System 

-Photo- 

20 

2.3 

.9 

9.5 

2.5 

Trainer 

based 

image 

generater 

-Mini 

displays 

38 

2.6 

1.2 

7.0 

2.4 

.9 

MULTI- 

Networking 

-Data  base 

12.5 

2.1 

1.2 

14.0 

2.6 

1.2 

PLAYER 

math 

EXERCS 

model 

develop 

-Data 

transfer 

12.5 

2.1 

1.2 

22.0 

2.8 

1.3 

-Environ 

45.5 

2.8 

1.1 

27.5 

2.9 

1.2 

params 

modeling 

-Network 

security 

60 

3.1 

1.0 

55.0 

3.5 

1.0 

VIRTUAL 

Visual 

-Head- 

2.5 

nm 

22.0 

2.8 

REALITY 

Displays 

mounted 

visual 

displays 

1 

1 

1 

Tactile 

-Force 

42 

50.0 

ID 

Displays 

simul 

■■■ 

mu 

mg 

■■■ 

Audio 

-Head- 

62 

3.2 

.9 

38.5 

3.2 

ID 

Displays 

mounted 

stereo 

m 

Eye  Mvmnt 
Transducer 

-Eye  track 

45.5 

2.8 

B 

55.0 

3.5 

D 

Key:  1  =  Very  Critical/Very  Knowledgeable 
3  =  Moderately  Critical/Knowledgeable 
5  =  Minimally  Critical/Limited  Knowledge 
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DELIVERY 

TRAINING 

ENABLING 

CRITICALITY 

COMPETENCE 

METHOD 

TECHNOLOGY 

TECHNOLOGY 

RANK 

RAT 

STD 

RANK 

RAT 

STD 

ING 

DEV 

ING 

DEV 

EMBEDDED 

Engineer- 

-Sys 

12.  S 

2.1 

B 

55.0 

3.5 

1.0 

TRAINING 

ing  Tech- 

safety 

nologies 

-Sys 

26.5 

2.4 

35.0 

3.1 

.9 

stimul 
-Sys  fail 
safe 

26.5 

2.4 

m 

55.0 

3.5 

.9 

-Sys  mode 
conversion 

45.5 

2.8 

D 

38.5 

3.2 

.9 

Automated 

-Intel 

26.5 

2.4 

1.1 

18.0 

2.7 

■■ 

Instruc- 

agents 

tional 

-Auto 

38.0 

2.6 

1.4 

45.0 

3.3 

Support 

perform 
meas  & 
feedbk 
-Missing 

45.5 

2.8 

1.1 

45.0 

3.3 

.8 

team  memb 
-Auto 

56.5 

3.0 

1.4 

55.0 

3.5 

adaptive 

instruct. 

HH 

Training 

-Lesson  & 

60.0 

3.1 

n 

62.0 

3.7 

.8 

and 

scenario 

Organiza- 

authoring 

bHH 

tional 

-Auto 

64.0 

3.6 

55.0 

3.5 

1.0 

Management 

record 
management 
-Coord  & 
standardi¬ 
zation 

65.0 

3.9 

B 

64.0 

3.8 

.9 

Key:  1  =  Very  Critical/Very  Knowledgeable 
3  =  Moderately  Critical/Knowledgeable 
5  =  Minimally  Critical/Limited  Knowledge 
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DELIVERY 

METHOD 


TRAINING 

TECH 


ENABLING 

TECH 


CRITICALITY  | 

RANK 

RAT 

ING 

STD 

DEV 

COMPETENCE 


BROAD 

DEVEL 

STRATGS 

-  TECHS 

TARGET 

MULTIPL 

DELIV 

METHODS/ 

STRATGS 

&  TRNG 

TECH 


Threat 

Modeling 

and 

Common 

Data 

Bases 
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16.0  2.2 
20.0  2.3 

20.0  2.3 


26.5  2.4  .8  9.5  2.5 


26.5  2.4 

32.5  2.5 


27.5  2.9  I  1.0 

18.0  2.7 


38.0  2.6  .8  27.5  2.9 


-Int 
agnt 
-Reusbl 
-Scenrio 
control 
-Knowldg 
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pedago¬ 
gical 
knowldg 
modeling 
-Knowldg 
acquis. 
-Infrnce 
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-Docmnt 
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50.0  2.9 


.6  45.0  3.3 


50.0  2.9  1.2  45.0  3.3 
56.0  3.0  1.0  62.0  3.7 


60.0  1.1 
63.0  3.3 


Key;  1  =  Very  Critical/Very  Knowledgeable 
3  =  Moderately  Critical/Knowledgeable 
5  =  Minimally  Critical/Liroited  Knowledge 


27 


Special  Report  93-005 
TABLE  5  (Cont.) 
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.9 
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.9 
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56.0 

3.0 
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3.0 
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62.0 
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62.0 

3.2 

.6 
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1.0 
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Key;  1  =  Very  Critical/Very  Knowledgeable 
3  =  Moderately  Critical/Knowledgeable 
5  =  Minimally  Critical/Limited  Knowledge 
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The  eight  Enabling  Technologies  with  the  highest  overall 
Criticality  ratings  (2.0  or  less)  are  listed  in  Table  6.  The  eight 
Enabling  Technologies  with  the  lowest  overall  Criticality  ratings 
(greater  than  3.0)  are  listed  in  Table  7.  The  14  Enabling 
Technologies  with  the  highest  overall  Criticality  ratings  among 
just  the  most  behaviorally  oriented  topics  are  shown  in  Table  8. 
Criticality  ratings  are  shown  for  both  the  total  group  of 
respondents  and  the  subgroup  of  behavioral  respondents,  along  with 
associated  Training  Technologies  and  Delivery  Methods. 


Table  6 

MOST  CRITICAL  ENABLING  TECHNOLOGIES 


TECHNOLOGY  CRITICALITY  RATING 


-  Reusable  software 

1.5 

(1.5)* 

(Broad  Development  Strategy) 
Interoperability/correlation 

1.5 

(2.5) 

(Threat  modeling  and  common  data  bases; 

Broad  Development  Strategy) 

-  Automated  photographic  data  base  generation 

1.7 

(2.5) 

(Visual/sensor  simulation; 

Broad  Development  Strategy) 

-  High-quality,  low-cost  image  generation 

1.8 

(3.0) 

(Visual/sensor  simulation; 

Broad  Development  Strategy) 

-  Software  development  methodology 

1.8 

(2.0) 

(Broad  Development  Strategy) 

-  Environmental  simulation 

1.9 

(2.0) 

(Broad  Development  Strategy) 

-  High  performance  computing 

2.0 

(2.0) 

(Broad  Development  Strategy) 

-  High  quality  lightweight  optics  for  HMD 

2.0 

(3.0) 

(Visual/sensor  simulation; 
Broad  Development  Strategy) 


*  The  ratings  in  parentheses  are  associated  only  with  the 
behavioral  respondents. 

Key:  1  =  Very  Critical;  3  =  Moderately  Critical;  5  =  Minimally 

Critical 
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All  eight  of  the  Enabling  Technologies  rated  most  Critical 
are  Broad  Development  Strategies  and  primarily  engineering  in 
nature  (Table  6) .  High  Criticality  found  for  the  Broad 
Development  Strategies  is  expected,  because  of  their  wide 
applicability  across  training  problems.  The  higher  ratings  for 


Table  7 

LEAST  CRITICAL  ENABLING  TECHNOLOGIES 


TECHNOLOGY 


CRITICALITY  RATING 


-  Network  security 

3.1 

(3.5)* 

(Networking;  Multi-player  Exercises) 

-  Documentation 

3.1 

(2.3) 

(Expert  systems; 

Broad  Development  Strategy) 

-  Enhanced  lesson  and  scenario  authoring 

3.1 

(3.0) 

(Training  and  organizational  management; 
Embedded  Training) 

-  Head-mounted  stereo 

3.2 

(3.0) 

(Audio  displays;  Virtual  Reality) 

-  Operations  research 

3.2 

(3.0) 

(Broad  Development  Strategy) 

-  Explanation 

3.3 

(3.0) 

(Expert  systems; 

Broad  Development  Strategy) 

-  Automated  record  management 

3.6 

(3.5) 

(Training  and  organizational  management; 
Embedded  Training) 

-  Coordination  and  standardization 

3.9 

(3.0) 

(Training  and  organizational  management; 
Embedded  Training) 


*The  ratings  in  parentheses  are  associated  only  with  the 
behavioral  respondents. 

Key:  1  =  Very  Critical;  3  =  Moderately  Critical;  5  =  Minimally 

Critical 


the  engineering  than  behavioral  technologies  is  expected  because, 
as  noted  earlier,  the  respondents  had  primarily  engineering 
backgrounds.  Differences  in  perceptions  of  Criticality  between 
behavioral  and  engineering  respondents  are  shown  by  the  generally 
lower  Criticality  ratings  assigned  to  these  items  by  the 
behavioral  respondents  (behavioral  respondents'  ratings  are  given 
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in  Table  6  in  Parentheses)  .  Only  two  of  the  eight  items  — 
Reusable  software  and  High  performance  computing  —  received 
ratings  from  behavioral  respondents  equal  to  the  overall  group; 
the  remaining  six  were  given  moderately  to  significantly  less 
Critical  ratings. 


The  Competence  ratings  for  the  eight  most  critical  Enabling 
Technologies  also  are  high:  five  are  among  the  top  six  in 
Competence  ratings  among  all  65  Enabling  Technologies,  and  all 
eight  have  ratings  within  the  upper  quadrant  of  the  Competence 
scale. 


TABLE  8 

MOST  CRITICAL  BEHAVIORAL  ENABLING  TECHNOLOGIES 


TECHNOLOGY  CRITICALITY  RATING 


-  Head-mounted,  low  cost  visual  displays 

2.1 

(3.0)* 

(Visual  displays;  Virtual  Reality) 

-  Teamwork  skills  management 

2.3 

(1.5) 

(Team  performance  measurement 
system;  Team  Training) 

-  Training  evaluation  measures 

2.3 

(1.5) 

(Team  performance  measurement 
system;  Team  Training) 

-  Intelligent  agents 

2.3 

(1.7) 

(Expert  systems;  Broad  Development  Strategy) 

-  Intelligent  agents 

2.4 

(2.0) 

(Automated  instructional  support; 

Embedded  Training) 

-  User  interface 

2.4 

(2.5) 

(Threat  modeling  and  common  data  bases; 

Broad  Development  Strategy) 

-  Exercise  development 

2.5 

(2.0) 

(Team  training  system;  Team  Training) 

-  Team  training  strategies 

2.5 

(1.5) 

(Team  training  system;  Team  Training) 

-  Scenario  control 

2.5 

(1.7) 

(Expert  systems;  Broad  Development  Strategy) 


♦Ratings  in  parentheses  are  associated  only  with  the  behavioral 
respondents. 

Key:  1  =  Very  Critical;  3  =  Moderately  Critical;  5  =  Minimally 

Critical 
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The  eight  Enabling  Technologies  with  the  lowest  Criticality 
ratings  (Table  7)  also  are  among  the  lowest  in  Competence  ratings, 
with  ranks  ranging  between  38.5  and  65  out  of  the  65  Enabling 
Technologies.  Lower  Competence  ratings  might  be  expected  for  the 
five  topics  that  relate  to  Expert  Systems  and  Embedded  Training, 
since  these  areas  were  defined  from  behavioral  science  viewpoints 
and,  again,  only  three  of  the  18  respondents  were  human  factors 
specialists.  Differences  in  perception  of  Criticality  between 
behavioral  and  engineering  respondents  are  shown  by  the  generally 
higher  Criticality  ratings  assigned  to  these  items  by  the 
behavioral  respondents  (behavioral  respondents'  ratings  are  given 
in  Table  7  in  parentheses) .  Only  one  of  the  nine  items.  Network 
security,  received  a  lower  rating  from  the  behavioral  respondents. 
But,  except  for  Documentation  (for  Expert  systems)  both  the  total 
group  of  respondents  and  the  behavioral  subgroup  rated  these  items 
lower  than  the  overall  average  for  Enabling  Technologies  (2.5). 
Once  again,  the  respondents'  competence  with  an  area  should  be 
considered  when  evaluating  Criticality  ratings. 

Enabling  Technologies  with  strong  behavioral  orientations  are 
treated  separately,  because  of  the  dominant  engineering  orien¬ 
tation  of  the  raters  and  the  low  Competence  ratings  assigned  to 
these  items.  The  nine  top  rated  behaviorally  oriented  Enabling 
Technologies  have  Criticality  ratings  ranging  between  2.1  and  2.5, 
and  mainly  support  the  Team  Training  and  Expert  Systems  Delivery 
Methods  (Table  8) .  Once  again,  differences  between  engineering 
and  behavioral  perceptions  are  noted,  with  the  behavioral  respon¬ 
dents  ratings  being  higher  on  seven  of  the  nine  items,  the  two 
exceptions  being  Head-mounted,  low  cost  visual  displays  and  User 
interface. 

VALIDITY  ISSUES 

Efforts  to  assess  potential  benefits  to  be  derived  from  new 
training  technologies,  such  as  the  current  survey,  are  limited  by 
a  lack  of  laboratory  and  field  evaluations  on  the  value  of  train¬ 
ing  technologies  in  general  and  the  new  technologies  in  partic¬ 
ular.  Such  evaluations  should  be  comprehensive  and  can  be  costly. 

Criteria  for  a  comprehensive  assessment  are  identified  by  the 
NAVPERESDEVCEN-NAVTRASYSCEN  Plan  (1992) .  The  criteria  are 
subsumed  under  four  categories:  (a)  "Foundations"  —  the 
fundamental  ability  of  the  technology  to  benefit  the  Navy;  (b) 
"Affordability  Factors"  —  the  costs  associated  with  realizing  the 
benefits;  (c)  "Training  Situation  Applicability"  —  the  range  of 
situations  where  the  technology  can  be  used;  and  (d)  "Risks"  — 
barriers  to  the  successful  implementation  of  the  technologies 
(Appendix  B)  .  Technology  surveys  should  consider  all  such 
factors,  but  a  suitable  data  base  upon  which  to  base  such 
considerations  has  yet  to  developed. 


32 


Special  Report  93-005 

CONCLUSIONS 


GENERAL  IMPLICATIONS 

This  survey  identifies  and  defines  91  training  technologies 
that  are  suitable  topics  for  military  R&D  programs.  The  major 
finding  from  the  survey  is  that  training  experts  —  high  level 
scientists  and  managers  from  private  industry  and  the  NAVTRASYSCEN 
—  consider  most,  if  not  all,  of  these  technologies  important 
topics  for  future  R&D.  A  review  of  related  efforts  to  define 
training  technology  priorities  for  R&D  showed  a  high  degree  of 
similarity  with  the  current  results. 

Additional  confirmation  for  the  priority  of  the  technologies 
is  the  fact  that  senior  research  personnel  at  the  NAVTRASYSCEN 
were  specifically  directed  to  identify  the  highest  priority  R&D 
issues;  and  the  training  experts  who  reviewed  and  critiqued  the 
survey  generally  agreed  that  the  survey  contained  the  highest 
priority  R&D  issues. 

The  current  survey  shows  high  priority  for  the  technologies 
by  the  finding  that,  across  all  91  survey  topics,  80  (88%)  were 
rated  as  Moderately  Critical  to  Very  Critical.  These  are 
considered  especially  high  ratings  because  of  the  nature  of  the 
rating  scale:  a  "Very  Critical”  rating  required  an  extremely  high 
evaluation,  which  would  tend  to  shift  all  ratings  toward  the  low 
end  of  the  scale. 

Behavioral  topics  were  expected  to  receive  lower  Criticality 
and  Competence  ratings  than  engineering  areas,  because  of  the 
predominantly  engineering  backgrounds  of  the  respondents. 
Although  these  expectations  were  confirmed,  behavioral  issues 
still  received  high  Criticality  ratings.  This  may  be  attributed 
to  both  the  broad  perspectives  of  the  raters  and  the  importance  of 
the  behavioral  topics.  As  expected,  behavioral  areas  received 
higher  ratings  from  the  behavioral  respondents. 

The  extent  to  which  the  respondents'  familiarity  with  an  area 
influenced  the  criticality  ratings  is  not  entirely  clear,  but  some 
such  influence  is  apparent.  This  suggests  that  the  overall  group 
priorities  given  to  the  behaviorally  oriented  Technologies  should 
be  considered  conservative  estimates,  at  best.  So,  the  current 
survey  is  viewed  as  a  fair  measure  on  engineering  issues,  but  a 
nonrepresentative,  and  probably  conservative,  measure  of  expert 
opinion  on  behavioral  topics.  For  that  reason,  efforts  to  make 
fine  priority  discriminations  based  on  the  ratings  may  be 
justified  for  engineering  areas,  but  are  probably  inappropriate 
for  the  behavioral  topics.  The  following  discussion  should  be 
considered  with  this  in  mind. 
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The  survey  could  be  improved  by  increasing  the  number  of 
raters,  and  particularly  increasing  the  percentage  of  raters  in 
behavioral  disciplines.  Then,  ratings  could  be  considered  across 
all  raters  for  all  areas;  but  the  most  valid  conclusions  might 
result  from  considering  the  ratings  only  where  Competence  is  high. 

The  survey  could  be  improved  by  offering  more  explicit 
criteria  upon  which  to  base  the  ratings,  such  as  identified  in  the 
NAVPERESDEVCEN-NAVTRASYSCEN  Plan  (Appendix  B) .  Surveys,  and  all 
other  efforts  to  assess  training  technology,  require  better 
information  from  systematic,  empirical,  performance-based 
evaluations  of  the  technologies.  Surveys,  such  as  the  current 
effort,  give  relatively  quick  answers  to  critical  questions,  and 
the  answers  may  be  the  best  available,  but  they  should  be 
continually  validated  with  the  more  objective  approaches  to 
support  a  training  science. 

DELIVERY  METHODS 

Team  Training  and  Deployed  Training,  tied  for  highest 
Criticality  ratings,  share  top  priority.  Multi-Player  Exercises, 
third  highest  in  Criticality,  is  the  third  highest  priority. 
Virtual  Reality  follows  with  fourth  priority.  The  Training 
Technologies  and  Enabling  Technologies  supporting  these  high 
ranking  Delivery  Methods  also  received  high  Criticality  ratings. 

The  high  Criticality  ratings  given  to  the  two  team  training 
topics  shows  strong  support  for  team  issues,  and  suggests  that 
highest  priority  should  be  given  to  team  issues  in  general. 

Emulation  and  Embedded  Training  tied  for  fifth  in  priority 
levels.  Interactive  Courseware  rated  seventh  in  priority.  All 
seven  of  these  Delivery  Methods  have  ratings  of  Moderately 
Critical  or  higher. 

The  remaining  two  Delivery  Methods  —  Wargaming  on  Ranges 
with  Operational  Equipment  and  Low  Cost  PC  Applications  — 
received  ratings  slightly  less  than  Moderately  Critical  and  tied 
for  lowest  priority. 

Interactive  Courseware  and  Low  Cost  PC  Applications,  ranked 
low  by  the  total  group,  ranked  high  in  the  ratings  from 
behaviorally  oriented  respondents.  This  suggests  that,  while 
engineering  issues  are  less  urgent  in  these  areas,  major 
behavioral  issues  remain. 

TRAINING  TECHNOLOGIES 

Training  Technologies  with  the  highest  Criticality  ratings 
also  support  the  highest  rated  Delivery  Methods,  or  else  give 
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broad  support  across  a  range  of  technical  areas. 

The  two  Training  Technologies  with  the  highest  Criticality 
ratings  are  Visual/ Sensor  Simulation  and  Carrier  Based  Weapon 
System  Trainer,  in  that  order.  Third  highest  is  shared  by  four 
Technologies:  Visual  Displays,  Networking,  Team  Performance 
Measurement  Systems  and  Team  Training  Systems. 

The  remaining  11  Training  Technologies  also  have  high 
Criticality  ratings  (above  Moderately  Critical) ,  that  recommend 
them  as  R&D  issues.  One  possible  exception  is  Training  and 
Organizational  Management;  but  this  is  a  behavioral  item,  and 
probably  not  appropriately  represented  by  the  data. 

As  expected,  the  behavioral  respondents  rated  behavioral 
areas  higher  than  the  total  group  ratings.  Team  training 
Technologies  clearly  lead  in  the  ratings,  followed  closely  by 
Expert  Systems  and  Automated  Scenario  Generation  and  Control. 

ENABLING  TECHNOLOGIES 

The  bias  in  the  sample  of  raters  appears  to  have  had  the 
strongest  influence  on  the  Enabling  Technologies.  All  eight 
Enabling  Technologies  shown  in  Table  6  have  strong  engineering 
orientations,  and  all  eight  rank  highest  in  Criticality  ratings 
from  among  the  65  Enabling  Technologies.  The  dominant  discipline 
is  less  obvious  for  some  of  the  Enabling  Technologies  rated  lowest 
in  Criticality,  but  six  of  the  eight  rated  lowest  were  defined 
with  strong  behavioral  perspectives.  The  two  exceptions  are 
Network  security  and  Operations  research  (Table  7) . 

Competence  ratings  also  are  lower  for  the  six  behavioral 
items  rated  lowest  in  Criticality  and  higher  for  the  eight 
engineering  items  rated  highest  in  CriticaTity.  Again, 
Criticality  ratings  for  the  behavioral  items  should  be  considered 
conservative  estimates,  at  best. 

Differences  in  perceptions  of  Criticality  between  engineering 
and  behavioral  respondents  were  shown  by  lower  Criticality  ratings 
from  the  latter  compared  with  overall  group  ratings  for  the  high 
rated  engineering  Enabling  Technologies;  and  higher  ratings  from 
the  behavioral  respondents  compared  with  overall  group  ratings  for 
both  high  and  low  rated  behavioral  Enabling  Technologies. 

A  large  majority  of  the  Enabling  Technologies  (57  of  65)  have 
Criticality  ratings  ranging  between  Very  Critical  and  Moderately 
Critical.  This  supports  the  conclusion  that  most  or  (considering 
the  sampling  bias)  all  survey  topics  are  high  priority  areas  for 
R&D. 
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All  eight  Enabling  Technologies  rated  highest  in  Criticality 
support  a  wide  range  of  technical  areas  (i.e.,  they  are  Broad 
Development  Strategies) . 

Behavioral  Enabling  Technologies  with  the  highest  Criticality 
ratings  (2.1  through  2.5),  while  lower  than  the  highest  rated 
engineering  counterparts  (1.5  through  2.0),  are  still  considered 
strongly  recommended  for  R&D. 

Enabling  Technologies  that  support  the  highest  rated  Delivery 
Methods  had  high  Criticality  ratings  themselves.  This  indicates 
that  these  Technologies  are  important  elements  of  the  Delivery 
Methods . 
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RECOMMENDATIONS 

o  Consider  all  technology  areas  identified  in  the  STTS  for 
development  and  implementation. 

o  Prioritize  selected  topics  for  R&D  based  on  findings 
presented  in  this  report  together  with  the  benefits,  costs,  risks, 
and  applicability  of  particular  technical  developments.  Consider 
criteria  such  as  described  in  the  NAVPERESDEVCEN-NAVTRASYSCEN  Plan 
(Appendix  B)  to  help  guide  this  process. 

o  Improve  Navy  research  and  test  facilities  to  better 
evaluate  promising  technologies. 

o  Improve  evaluations  of  fielded  training  systems  to  assure 
their  proper  use  and  cost  effectiveness. 
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COORDINATION 


The  STTS  began  when  Mr.  H.  Okraski  tasked  the  NAVTRASYSCEN 
Research  and  Engineering  Department's  Science  Advisory  Board  (SAB) 
to  identify  and  prioritize  future  training  technologies  and 
methodologies.  This  initial  effort  was  expanded  into  the  STTS  and 
distributed  to  other  NAVTRASYSCEN  personnel.  It  was  also 
distributed  to  personnel  from  private  industry  through  the 
auspices  of  the  National  Security  Industry  Association  (NSIA)  R&D 
Subcommittee  (POC:  Dr.  James  L.  Davis)  and  the  American  Defense 
Preparedness  Association  (POC:  Art  Banman) .  Persons  who 
developed,  reviewed,  and  responded  to  the  STTS  are  listed  in 
Appendix  D. 
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APPSMDIZ  A 


TECHNOLOGIES  IDENTIFIED  AND  RECOMMENDED  FOR  FUNDING 
IN  THE  TRAINING-2000  REPORT 


4 . 0  PROMXSIMC  TBCHNOLOCIES 


Th«  slBulator  industry  takss  advsntsgs  of  axistlna,  now  and 
•■•ffing  tachnologlas  In  ths  daslgn  of  training  aystaas. 

Industry  la  strongly  ancouragad  to  usa  conarcially  availabla 
hardwara  and  softvara  in  an  affort  to  ■iniaita  Ufa  cycla  costs. 
Th*ra  ara  savaral  induatry-vida  tachnologlas  that  ara  raady  for 

training.  Tha  task  fores  has 

idantifiad  tha  following: 


IMWSTRY-WIDE  TBCHMOLOCiea 

a.  Artificial  Intalllganca/Expart  Systaas 

b.  Zaprovad  Visual  Displays  (Out-Tha-Nindow,  Halaat-Mountad. 
Pro j action  Systaas  and  High  Daflnltion  Talavlslon) 

c.  Modular/Rausabla  Softwara  and  Hardwara 

d .  Softwara  Tools 

a.  Advancad  Cosgmtar  Archltacturs 

f.  Low-Cost  Coaputars 

g.  Oatabasa  Ganaratlon/Corralation/Puslon 

h.  Ratworking 

1 .  Eibar  Optics 

j.  Talacoaaunication  Satallitss 


What  follows  ara  training  slaulator-paculiar  tachnologlas  that 
ara  raady  for  axploltatlon  through  appllad  rasaarch  and 
daaonstratlon. 


gimiATQR  TEOMOLOGIES 

a .  Thraat  Modals 

b.  Babaddad  Training  Signal  Ganaratlon 

e.  Autoaatsd  ISO  Tools/Docuaantatlon 
d.  Scanarlo  Production  Systaas 

a.  Autosatad  Trainaa  Parforaanca  Msasuraaant 
t.  Training  Effactlvanass  Assassaant  Tools 

g.  Taaa  Training  Tachnology 

h.  Rapid  visual  Oatabasa  Construction/Nodlflcatlon 
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TABLE  1 

Recommended  Research  Issues  and  Required  Funding 


Research  Issue 

Rtaaircl 

■Fuading 

Build  Navy  Training  Requirements  Simulator 

$8 

M/yr. 

Expand  Research  for  Mission  Planning,  Mission 
Rehearsal,  Team  Training 

$6 

M/yr. 

Expand  Research  for  Embedded  Training 

$s 

M/yr. 

Expand  Research  for  Visual  Imagery 
and  Weapons  Effects 

$5 

M/yr. 

Expand  Research  for  Computer-Based 

Classroom  Training 

$3 

M/yr. 

Expand  Research  for  Instructor/Operator 

Systems  and  Standard  Software 

$4 

M/yr. 

A-2 


Special  Report  93-005 

APPEMDIZ  B 

MAJOR  CONCLUSIONS  FROM  THE  NAVPERSDEVCEN-NAVTRASYSCEN  PLAN 


Table  I 


Connections  among  Trends,  Training  Requirements, 
and  Training  Technologlee. 


Trend 

Change  in  missions 

Change  in  weapons  technology 

Jointness 

Quick-response  requirement 

Reconstitution 

Are  we  ready? 

Is  training  effective? 

Fewer  People 

Reduce  shore  Infrastructure 

Fe'  “r  Dollars 

More  complicated  systems 
New  information  displays 
Deficits  in  Entering  Skills 


Requirement 

Train  for  new  missions... 
..against  new  weapons... 
...wMi  other  Services 
What  to  train? 

Develop  simulations  rapidly. 
Denver  training  rapidly 
Counter  skill  degradation 
Up-to-date  training  content 

Scale  up  rapktiy 


Measure  proficiency... 

...and  knowtedge 

Widely  available  cross-training 

Reduce  skill/knowiedge  loss 

Develop  and  deliver... 
...training  more  efficiently 


Re-use  training 

Improve  training  management 
More  effective  training 


Increase  understrmding 


Include  In  training 


Technology 

Warfara/Tactics  Simulation 
Platform  ahnuiallon 
Meroparacnty 
Task/oognllive  analysis 

Authoring  aUn 
Networttt/DUrtbuled  training 
Deployable  ahnuiallon 
Authoring  Alda/Cuntculum  rtatahan^ 

Networks/Computer  delivery 
Distributed  training 
Management  Tools 

Performance  assessment 
Cognitive  assessment 

Nelworka/DisWbuted  training 
Computer-delivary 
CogniUvs  analysis 

Authoring  aids 
Nelworks/DisMxjted  training 
Improved  Instructional  Processes 

Networks/Computer  delivery 
Curriculum  Databases 
Nelworks/Management  Toots 
Cognitive  analysis 
Context-based  training 

Cognitive  analysis 
Context-based  training 

Simulation 
New  delivery  media 

Cognitive  analysis 
New  deWery  media 


Where  to  start? 
Counter  low  literacy 
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TabtoK 

Investmwit  CrttMla 


KMINMnONi 

MMuriiy:  'Dm  MOM  IB  «Mcti  Mehnelogr  MM  bt  taipl*- 
NWMad.  Mob  MMiuilir  IndtoaM  M  iwortuwi  and 

fiNM  iQf  ■npvHwnwvn  w  wh  (mvm  ■■ 
tfmaiM/dtMkm  tor  daliwiy  f»  wwieia.  LMiMdurty 
(idieatM  dial  dia  pracadmaa  tn  noi  Iliad,  imI  dadnad. 


Raaaarak  aaaa:  bidlBaiaa  dM  aniauM  ol  aatavai*  auppaidag 
caaaaiBh.  Fat  a  *hiah*  ndno  aiany  aludiaa  and  airn* 
diaaia  01  diad  taauia  IndlcMi  hao  and  why  dia  lachnab 
aoramka. 

R40  caln:  AmouM  ol  gain  (uneatlalniir  laduedon)  aigiinlad 
Inm  taaaaich.  A  *higti*  ndng  MteaMa  din  aueeoaM 
RAD  amiW  tncramani  aaldaoea  aboM  dw  laehMlaay 
daaipn  eaneapt  aubaianddbr- 

MiKary  -Uwlquanaaa*:  Oograa  ta  adiieh  dia  laehnelady  la 
apadfe  at  etWeal  te  dM  oillaty,  ae  tM  davotapmani 
aheuM  ba  aecalataiad.  Tadwaleglaa  iRh  loiga  eom- 
mortial  markou  «■  ba  daaalopad  ragatdiaaa  tt  Navy 
funding. 


AFFORDABILITY  FACTOm 

RAD  Caafa:  Higti  •NlatdabBiy*  didlaNaa  dial  dia  RAO  naadad 
la  davolag  tha  taohnigiia  fat  ptaedeal  uoa  la  taMNaly 
bMBpanoiva.  TMi  b  a  iaeior  kbpatiaid  tot  daeUbig 
wfiodiat  ta  ouppati  vaModon  damanatraton  ol  dia  loeli- 
nobgy.  Xew  aflordaWRy*  todtoRaa  dM  eaat  «auU  ba 
Ngh. 

Oavatopmonl  Ooaia:  AllaadabBto  of  davatogmaM  ratora  ta  dw 
dia  aaoia  of  davatoping  cRangoa  ofidM  addktotia  IB  dia 
oxialing  dafeibig  ta  ImptoiitanI  dw  tadinatogy.  TNa 
inebdaa  davotopmaw  of  tavbad  tfaMnp  twad  tridi  dw 
madwd  at  tachnatogy.  and  awuW  narmoly  ba  fundod 
iMiOAM. 

Copbal  Oaalr.  Ratoia  la  dw  anadnw  cwdM  Invaaimani 
laquirad  ta  bnplamanl  dw  mollwd  at  locfinatagy.  IMa 
awuid  nemnaly  ba  fundad  bom  aptoouramari  ooeaunL 

Lira  Cycto  Coaaa:  Tlw  obBiy  of  dw  ayatam  la  bow  dw  coal  of 
mainuwing  dw  cutroncy  of  dw  dovotopad  tochnotogy  to 

proaarva  dw  biUal  invaainwnL 


TTUmHO  SITUATION  AFPUCABILITV 
Raawatidal  bwirualton;  Uoolulnaaa  of  dw  wefmetogy  tot  prt- 
maty  inatiwaton.  How  uaWid  onuld  I  ba  tot  main-tow 
canvoniianal  training. 

Dtolribytad  Trabibig:  Uaaluirwaa  of  dw  taefinatogy  tor  training 
dialributad  to  romota  tocadona,  ouch  aa  dapioyad  par- 
aennol.  or  raiarviali 


Waiiwdtol  bwbtiBiton;  ButaUBy  tor  rapaMnp  dolidu  in  aiu- 
^atdnanlal  iIrtBa;  Uaatobiaaa  al  dw  todwatogy  tot  lunda- 

wMiWi  WB  WWp*  njnBBnIBnv  MB  BTO  mOBB  BIS 

SB  MBunitd  Navy  MninQ  iMtBrMi  and  Iota,  Mud- 
tor-  Mto  naadad  to  itndatalatid  Mat  and  nwnuato;  Ndb 
MBdidtofeld  Bfid  dMntaB  MomiiMn*  iMIb  iiBBdBd  Id 

a^^^^R^^np  Md  ^nna  wiMMyai^Hlldada  ^^dHw  iM^^wd^na  dv 

ibBa  naadad  la  poitotm  ondw^ab, 

MMaiy  Ortoittaltoii!  BudMMy  ol  dw  ladmatogy  tor  tawBart- 
ANton  at  ottonlNton  training. 


OanUmibig  | 

tibdKytotaondnuIngi 


:Sut- 


VvbMmii  Solvlnss  UmIuImbb  ot  ihB  tBdwolDQy  tof  Bnhwidi^ 


Ttanolar  la  Rani  Wntid!  UwBioad  dM  vdM  la  toamod  wB 
tranotortolototoaka. 

Qroug  partaanwnaa:  Uaabdnaai  ol  dw  laetinotogy  tor  group 
or  taam  dabiinB.  Can  dw  taefinatogy  ba  uood  to 
iniagraia  akBad  pattotmanoa  ■  taam,  uniL  boMa-giaup, 
totea,  at  thaatar  iBvaia. 

WdPPavatopiitanlrAppIraliBbicItacfinctogytoaMppodptac- 
daa  to  hoiw  akB  and  dov^  Ymntorto  adga*. 

ShBI  Mabitoiwiwa!  UHhy  el  dw  tachnotogy  tot  ladudng  pail^ 
atwbBRfcd  MB  during  paitedaaInBn-uaa. 


ta  BlaaBvanaaa  Maaauraiiiatif;  Can  dw 
affocla  el  dw  tachnatogy  ba  datomiinad.  DadwraoxiM 
aubdbtomaaautaa  el  aWacdvanaaadMdw  tachnatogy  la 


Aceaptob»>y;FeBlcaiaeBap<abB».IMwdwtdwtrtoningaya- 
lomwaiddtoatordwehaiigaeiiaaiatotlgnatalLakMi- 
ing  b  to  dto.  Da  apenaeta  andtor  daoaion  mahaw 
opprova  dw  tachnotogy  In  apha  el  ceolT 
Maownliim:  Patoalirod  andwatoom  tor  ImpianwnMian  by  dw 
itodf,  tor  Mtol  babilnB,  chongoa  to  phyatoai  piam.  acquir¬ 
ing  hardwata,  ole.  A  Yaw*  radnp  kiciudaa  parcoivad 
changoa  to  cenlanL  atoMng,  and  phyoical  ptord  dM 
aaam  too  hard  to  aacampdah. 

^ebohldly  el  eieeaoa:  Tha  probabBly  ifM  ImpianwnMian  of 
dw  tachnatogy  wfl  auocaaafuOy  achiava  dw  goah  aol  tor 


Buotobioblllly:  Whathar  or  not  dw  tachnotogy,  onca  knpla- 
nwMod  In  the  Navy,  would  ba  hapi  running  widioul 
aignlicaid  change,  or  ghadwt  rograaaion  to  oldar  aya- 
lama  it  Bwly. 
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Table  Ul 

Training  Syatams  Technologlaa 


Nmt:  Tht  utltukiglm  an  irtftd  ine  0tt  wefciatnio. 

narmioMf  fraccua  aW  mmMi,  OTrf  aWpMav  McAnetmiu. 


OEUVERY  TCCHNOLOOES: 

Pip«r-aH*d  iMkiMlIan:  TaehniquM  is  knpioM  pspw- 
orisnMd  MrudieMl  (Mlsfiali  Is  knprM  iMir  sMsciws- 
MM  in  iMMtfiowlidpn  noQuWtton.  ExMiplM  mt  twt 
tomai.  sddHsn  d  oebr  or  grapMet.  MlOf^  rssdSbay 
teniHilM  in  tnxt.  nddMont  of  liM^nQi,  tlwipHyinQ  tMl. 
impnving  sobsm  Is  idonnation,  ale.  TMi  araa  d  Mdt- 
ndogy  kidudas  wing  ooffipuMf  pmofaiM  to  appiy  ttMSS 
taehniquw  automatkaly. 

Orm  A  ftadtoa  Syatama:  Rapaatod  praaantotion  d  quaaliana 
poianiialy  aooenipsnisd  by  to4ia-laarnad  fflatoiid,  afton 
wad  in  cireumtatoad  dsmaina  wtti  toiga  numban  d 
facto  or  caleuldians.  Uauaiy  eempulaf  mariiitoii. 

Frama-basad  CM:  Pisaantation  d  taxi,  graphica,  or  ddao 
aeraana  d  to-ba^amad  malarial,  aridi  flw  iaamar 
advancing  in  tha  malarial  to  pointo  arttora  Mat  lama  may 
ba  prwanlad.  Framabaaad  C8I  ia  antiraly  pra- 
pragramoiad  during  instrueiional  davaiopmanl. 

VMao  Camaa:  Syaiama  buil  to  provida  pradica  anviionmania 
tar  datodion,  mator-aUI,  davioa-oonird  Maka,  and  tar 
axptoralan  d  udamKar  anvbonmada.  Olhar  laaming 
toaka  can  ba  ambaddad  in  lha  ganM. 

Scanarta-Eawd  Mmulattoiw:  Seanario-Baaad  Simuldiona 
ara  dian  but  nd  akwaya  Impkimaniad  on  oompuiar  aya- 
lama  Tha  aosnaito  pravidw  tha  ooniaxt  tar  lha  laarnar 
to  partorm  a  mm  d  adtona  to  aooompM  a 
laakAniatiorYgoal.  Untoa  tnaaplay  diaocMory  acanarioa 
arauaualyguidad. 

iCAl:  Inialigam  Comptfar  Aidad  Inalrudion.  in  oonlraal  to 
mamoriution  appreadiw,  aludanto  aaptora  and  kdarad 
with  an  kwirudional  program  highV  tatorad  to  a 
knowladga  baw  as  that  aludanto  ara  diarad  moia 
axianaiva  laadbadt  and  guWaneo  In  laaming  a  oompiax 
domain.  Piograma  tor  gonaraling  inalrudianal  krtarac- 
lion,  rathar  than  pra-programmad  inairuclion.  ara 
inekidod. 

Intaradivs  VMao:  Standard  IV  olian  Invdvaa  praasnialton  d 
aaquaneaa  from  a  vidao  diak,  CO-ROM,  or  olhar  doviea 
to  aeoompany  or  anhanoa  a  eomputar  baaad  tnatnclion 
program.  Emarging  anhanoamanto  kivoiva  aocaaaing 
larga  amotima  d  tad  or  graphica  atarad  on  tha  madia, 
dIan  with  datahaaa  laaturaa  allowing  etoai  linking  d 
kdonnalion. 

Equipmant  SknulatloiN:  A  phyataal  oaviea  ia  lapraaanlad  by 
eomputar  program!  daaignad  to  train  an  individual  hew 
to  oparata  or  maintain  dia  davioo.  Tha  laaming  iranalar 
d  tWBa  and  ooneapto  from  tha  aimdalad  to  tha  aetud 
daviea  may  or  may  not  raguira  high  fidaMly  dapandiig 
upon  tha  focua  d  lha  training. 


MuWtiaaf  SbnulaSan;  Sanw  w  pravloua,  axoapt  aavard  bidi- 
vMuali  wanakaorliadoonaotaatolniataetwkhgwtknu- 
Idton  oattoufiatdy. 

Oaneaplud  MaulaMsii:  Sbndaltona  wkhoui  hlghtowl!  d 
BiwifeilidiBv  taitf  wMeli  nmMfii  or  omoem 

M  oonoiftMl  or  Iwili* 

Eritoaddad  IMibig:  Training  pfovMad  by  sknulalion  wtotor 
dkmdagon  oapabMtoa  buil  bde  or  addad  onto  opara- 
ttond  aydama  and  cwnlaa. 

Plalfefiii  WiiiulatetafTlalhlng  Davteaa:  Thaaa  ara  typlr.ag> 
hardwara  davlew  did  ara  dwignari  is  bain  aapliok  )ab- 
laldid  DbddkiW.  Sknulaliena  aio  bdl  to  mimle  epara- 
ttond  ptadomN,  but  wih  vary  high  vlaud  and  physical 
gddhy,  a.g.  eghl  aiaiultfets. 

Vblual  Envbowwanta;  Uw  d  kitaraeiiva  oompuiar  graphica 
and  odiar  oempuUng.  dtopiay,  and  aanaor  tarhnniogiaa 
to  crada  tha  Bwion  d  baing  and  oparaling  in  a  dWarani 
anvironmanL  Tha  tochndogy  usaa  a  haad-mounlad 
vlaual  display  sdh  dllsrsnl  vtawa  tor  sach  ays  for  3-0 
vtawing,  aiarao  sound,  and  glovas  sritti  ambaddad  san- 
aors  to  oomrart  hand  modon  Mo  eomputar  oortooL 

Etodrenie  atoaaraom:  escbonie  olaasiooms  era  Magrmad 
nnlacdona  d  davlew  did,  to  varying  dsgraaa,  provida 
davstapmard.  praianidlnn.  and  managamars  d  kwbve- 
Itand  ffldsrfds  to  groupa  d  dudarta.  Capabiky  aaida 
lor  piaaandng  Iw  mdarldi  In  a  wrldy  d  madta  bnada. 
lading  tha  dudania,  and  provkSng  aummarizad  kdonna- 
don  to  dw  bwiiuctor  on  a  raaMbna  baah. 

Vlrtud  dsaaroom  An  alaoiionie  *eiasaioom'  w  dascribad 
Mova,  axoapi  sludanis  and  kwtnxaors  ara  not  physicaly 
eo-lecalsd.  biatoad  ihay  ara  connaciad  via  dda  nd- 
WOfItO* 

Dlatrlbutad  bitoradhra  Sbnulatien:  Pladofm  aknuldois  (and 
paihapa  opardional  piadoima  wkh  ambaddad  simuldion 
eapdiilhr)  Inkad  via  dda  nalwoika  wUi  aach  olhar.  so 
thd  savard  Indhiduals  or  loams  can  ba  trainad  in  muM- 
pldtorm  Maradloa  May  ba  appOad  d  variow  lavah  of 
aggrsgdton,  o.g.  erow,  pidlorm,  group,  torw,  thaalar. 


MSTRUenONAL  PROCESSES  AND  METNOOS: 
biatniotar  Ttabilng:  Tachniquas  to  Improva  lha  daWory  d 
eoursa  oonianl.  eorwnl  knowladga,  managamani  skils 
and  lading  tklla  In  lha  olaaaioom.  and  tachniquw  tor 
onhancingIndiuctorMSiyiohandlaona-on-onaMarae- 
don.  Including  diagnoalle  skila,  tutoring, 
btattiider  Evshialtan:  Tsehniquaadydsm  to  anharco 
kidiuclor  taaching  ddh  by  providing  ayslamdic  fsod- 
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back,  ragaidinQ  iMChing  partormann.  (ram  ahidaiaa 
and  tiaininQ  niaiiaQaia> 

TCM/TQII:  Aaatotpfadtoaaandpioeaaaana^malhadaiD 
ooftfinuoufiy  bnpiov#  iht  Md  ptoduolMy  4f 

BChOOihOtHS  TtM  lOQMB  1b  IRMH  HMMlQMIOTl  to 

Blill  on  oowlinuouB  toipwvofiionL  CxBmlnlnQ  dMouHno 
fn  MudOTi  toamlng.  Mutfoni  ttvoughpul.  and  pnvttng 
oofFooUvo  toodbook  to  InBiuoton. 

Studonl  tBomkig  Hritogtoo:  ToGhniqtiM  tor  onhoncino 
baoie  aludy  aUto.  auch  aa.  ooneantraiion  managamanl. 
aludy  tona  aianaQaiitont  mtotoklng  and  lalaninQ, 
aiamefy  aUa,  and  taal  MHng. 

OognWM  taambig  tfalaglaai  Tiainina  ki  watarwgidMiw 
aifUaqlai  let  anoodino,  raotganWno.  man^iuMins  Mar- 
airfan  <a  Incraaaa  ka  udKy  In  pwMam  aahdng.  or  la 
manwriMkf.  ar  iwnalaf  wdua. 

Prograaa  TaoUng:  Taating  ta  monitor  atudanTa  partotmanea 
tolar  oaeh  unt  or  aub-unk  to  InMnrellon.  Pmgcaaa  Mat- 
ing  Muady  la  oonoamad  todi  dadartolim  knowtodga. 

OtognoaMc  Tatobig:  Tatoing  ftto  mapa  tha  MudanTa  parlor* 
manea  onto  Sm  pmoadwal,  oonBapbito  and  knawladga 
goaia  to  tha  training  to  dlao^r  inatnietionto  or  iaaming 
ahoRMa. 

Adapllva  Taaling:  Tatoing  boaad  on  aadmalaa  to  lam 
dlfieuly  aooording  to  vattoua  matfwmaheto  madala. 
Uaualy  adffliniaiatad  by  campuiar. 

OaoparaUva  Uambig:  CoepataUira  Iaaming  la  dw  Inalrue- 
bonto  tortoagy  thto  haa  mom  than  ona  abidani  wwkino 
togathar  tor  totM  or  knawladga  aoquiakion.  TMa  can  bo 
aoeampliahad  «toh  or  •Wnul  automaiad  aida,  dapanding 
enihaaaaing.  lnNmtyantoRinmatoa,ooopartoivolaam* 
.  ktg  haa  baan  uaad  allaellM^  by  hatong  2  or  3  atudania 
working  on  tha  aama  malarito  to  ana  oompular  lanninaL 

Poor  Taatoilng:  Augmanta  Via  kiainictor  tmugh  Via  two  to 
atodania  to  aMha  lavala  to  Iaaming  to  tooch  oath  olhar 
in  onatoitona  tohialiank  Tha  naad  to  baeoma  tha 
inainietor  jntraaaaa  ahidant  metlvalion  «td  tha  numbar 
w  lOpoQoona  si  vw  laaaon. 

Coal  aatang:  Goto  atoling  fervolvaa  atoaMahlng  apaolic  cii- 
laria  loi  langih  to  abidy  or  praellea  or  la¥to  to  parfor- 
manoa.  Thoy  ean  ba  appiiad  feidMdualy  or  to  groapa 
(aaa  eoopartowo  teaming).  Ellaciiimnaaa  te  anhanoad 
whan  goaia  ara  ato  appropriately  lor  oaeh  aUidanL 

Extrbiaie  Molhmlten:  Exvinaie  moUvadonto  tatomlquaa  pto- 
vkte  rakrloreamato  tiaa  ia  aaiamto  to  tha  kiainielional 
procaaaroontanL  Eiamplaa  ara  Ikno  oH  bom  iraMng.  a 
promolion  lor  good  parlormanea.  a  ptequa  tor  baing  boat 
ahidani,  ehoioa  to  duty  aaaignmani  ato. 

CompotUon:  Compadtion  ia  a  mtokrtoionto  taehnteua  Vial 
may  ba  buto  Inao  a  aludy  taak  on  atoiar  an  individual  or 
group  baate.  Thia  alralagy  uaualy  Invalvaa  a  aoora 
baaad  on  iim»4»«)mpteia  andtor  parformanca. 

Contait-Baaad  Training;  Conlaal-baaad  trtorrkrg  ktooiportoaa 
avartetytotochntouatandmtohodaditoallnvtovabain- 
kig  in  a  job  Hu  anvironmanL  Tha  tachnology  aaaumaa 
that  teaming  ia  anhancad  whan  kiamrcaian  loouaaa  on 


dra  apaelte  abuadon  In  whieh  parloimanoa  te  to  oeeur. 
Thte  tokara  dra  toudard  to  team  by  doing  and  undar- 
toanitng  dw  purpoaa  to  dra  parderiter  laak  to  hand  in 
ntedon  to  dra  Moral  goto  to  htobuedoiL  Thua,diatou- 
dard  teama  to  pto  dra  telormadoA  laelA  tobitolnna.  and 

Soma  to  dra  lachrdguaa  haM  boon  uaad  In  aduetoionto 
tobitoionawhaiadiomtonoapaetoelab.  In  draao  oaaaa, 
diatoialagghaabaantDambaddiolnabuodonIniaala- 
dc  (rato  world)  ptohlama  or  caaaa.  Cotdoal  baoad  dton- 
teg  oonbaala  rtodi  arora  tradMorrto  kralrucdon  rtodeh  lypi- 
cily  laouaas  on  tonnal.  Qanofii  knowMoa  and  aUb 
ahatraetad  twm  thbr  uaaa  and  tough!  m  laolba  J  tapica 
and  ndaa.  Oeidato-baaad  dainhrg  lonna  dra  baate  to 
fliirw  aufiaiO  iMlitiiakMal  ladMlaiiaa  Indudho: 

daa^Mv^  dMadfwab  M^btodaMaMwaaitoto 

anohorod  braaruodorv  alhiaiad  oogrriion,  oogrtokm 
approndoaahlpb  harcdonto  oonlaid  dolirlngt  arork  arodate. 
and  dra  eaaa  aludy  madrod.  (Neaa:  mSlaiy  tachnleto 
raaaaith  haa  ahown  oontoto-baoad  training  to  motto  In 
lowar  allidlen  and  ■Wbark  talaa.  roduoad  daMng  tkna, 
auparior  orrtora'job  achlavainarA  ratardioru  and 

draaplay  Dtecoirary:  Fraaplay  Dteoovaiy  promoiaa  teaminp 
by  olowinQ  atudonia  lo  fiooly  oxplof a  an  an^lioiananl  la 
dteoovar  dra  temahutea^rinclptea  dial  gouam  A  Siu* 
danla  team  cauaa  and  altaci  rolalionahipa  by  mank^- 
feig  vaitihtea  In  dm  amdronmato.  MoM  to  tha  anviron- 
manta  dial  hava  boon  davalop  by  raaaarehara  hava  baan 
bnplamard  on  oompuiar  ayatorrra>  Soma  armmptea. 
krcluda  mioioatorlda  lor  artonomica  and  atecdiedy 

davatopad  to  LROC,  aa  wal  aa  Frodartchaon  and  WMtaa' 
- »-  1— .  -» — ^ 

OTin  toi  macpvsoy. 

Manlol  Modal  iiiMln0:  MM«b  Modal  BuMng  la  dm  taeh* 
niouo  M  nravidbn  otudofOo  immi  M  OtMUna  iboia 

d^o  adowto^^^w^^to  d^o  Mww^m^ai^p  iMiMimdb 

ayawmatoavloaatenvbonmarda  dial  totow  dram  todhg* 
noaa,  pradiei,  aapteki,  and  raoonabueL  Studarda  team 
nMMiJri  nodala  dvouQh  onatoQloa.  alndufal  and  lunc 
tionto  aiplantolona  and  aaairiptea.  bdormto  guatoadva 
raaaonkig  dtonfeip,  knagary  laehniquaa.  and  aaptoradon 
to  actual  andtor  akniitoad  cyatama.  daviooa,  or  anvkon- 

taartrktg  Stylaa;  Thte  lachniqua  kivtovoa  aalacdng  todaraid 
kratniedonto  approatowiAiaivary  madwda  baaad  on  a 
MudanTa  krdMduto  teaming  atyta.  For  aicampte.  ona 
poiaon  may  team  mom  aHacdvaly  liom  a  vidae  pmaan- 
ttolon  than  bom  a  taa  pnaordtoton. 

Mellvationto  Otembig:  Laauning  caat  in  a  oompadtkm  term  to 
aooumulMing  pokda  or  ^aitong  tha  dock*  to  achteva 
oormd  partormanoa,  olten  wMi  nova!  kdarlaoaa  that 
akndtoa  or  mpraiaid  a  teamteg  domain,  aomadmaa  in 
fardaay  term.  Gaming  lachniquaa  can  ba  uaad  to  add 
mdivadonto  variabtea  to  bama-  or  diS-lypa  kiiiniedon  to 
knprova  kdarato  and  pamWanoa  In  teaming. 

OvarlaamIngrAulomallelty:  Thte  tortoagy  ia  whara  rapatoad 
teaming  adKrWaa  am  providad  auch  dial  tha  toudard 
Vvariaama*  tha  akil  to  a  high  dagraa  and  tha  teamad 
bahavtor  baoomaa  automtoic,  givan  tha  appropriau 
adfflulua  aatdng.  Thto  la.  tha  atudam  car  parlorm  tha 
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activity  vfllh  vwy  M*  or  no  eogntivw  biMlMHiM. 
Moniial  foaoeiibig:  PraUtm-aoMng  booad  on  eoiOaiclMl 
knowMgo  an  ewoMd  Is  loimal  oyoMaa.  Lo«nar 
oaptoroaMpaitaneaa  0  ptoUani  tkMlian  taWi  appiaprt- 


8UPPOIITWO  TBCItlOLOaM. 

CoonlUM  AimImIk  DMfen/ldnMl^MMit  ^  ImlrudkHifll 
MarvanHoM  vio  taak  anotraia  pmeaduraa  aMch  bMOlvo 
apaoPyinQ  ttia  ooQitMoa  of  aiaittal  ovania  oi  paooaaaaa 
aaidonta  nwat  aequhou  raitat  than  niafo  obaantaMa 
bahaviofa.  InohiOaa  apod^fbiQ  maiaal  inodala  Of  anal^ 
giaa  apprapriala  for  kiabuciion.  Tho  preeaaa  bidudaa 
loGhniquoa  for  ctiaraeiatWng  bia  bnoailtdqahMi  ol  lab 
aipana,  roatoaaminy  a  bi  aoma  daaeilplian  languaga. 
anddaaoMng  highar  laval  abuelura  among  knoadadga 
atamantak 

Cegnblva  haaaaamant!  Tadmiquaa  far  daiaimining  tw 
Oagraa  la  oMeh  paopla  hava  aequirad  paitleular  aogni- 

tfld  pfOOMSMv 

Analyala  ol  OlaMbiiUd  CagnMan:  Piaoaaaaa  lot  charaetar- 
iaing  tM  nalura  ol  eognMvt  paifonnanea  an  taaka 
apraad  avar  gioupa  of  paogla  and  odiar  eoitipulaUonal 
davioaa.  bidudaa  taduiiquaa  tor  wpraaanting  tm  bdot- 
maiion  imartooaa  among  paapto  and  davtoaa.  iha  atabd- 
by  d  pattormanea  evar  tana,  raduttdartey,  mbuauiau 
imdar  uneartainty.  and  anar  raaavary. 

TaamOaetotofi  Making:  Chatadattoaltonoflima^aaaddad- 
aion  mddng,  kidudbig  dadaian  making  undar  airaaa. 

Analyatodadmalton  af  Taant  tahavtor:  bidudaa  aoifc  an 
laadoraMp,  mtoHitedairg,  taam  organbalton.  oanqMar- 
aupponad  oaopaiallva  lak.  eantomndng. 

Automaiad  AMa  for  Oaaignbig  bialrueiton:  Contouiariiad 
loda  and  dadaton  aida  tor  daaignbig  toantidtond  aya- 
i«a.  bidudaa  tank  anaiyaii.  madia  aatodton.  ale. 
Indudaa  ddabaaa  toob  tor  mainiaining  intoiienaNpa 
batwaan  babdng  gaato,  toabuciiond  avania,  partor* 

manGamaaauraoiania.bainingaqubnianl.ato. 

C8I  Aulfiorbig  Toato/Tamptolaa:  Proddtoad  aiabudtonal 
abatagiaa  thd  aeoapi  ooniani  tram  kiabudor  and  gon- 

arm  ttaining  bdaractiona  •Ml  atudama. 

Slmu^  Aulbarbig;  Highar  laval  languagaa«yitomamids 
tar  programming  aimulationa. 

Seanarto  Authoring:  Highar-lavd  languagaa/loola  lor  buddbrn 
aeanarioa.  atpadaly  tor  loam  training  and  parlormanoa 
aaaaaamanL 

Automatad  Parlorniaiioa  Haaauramant:  Indudaa  lach- 

raoording  paiformanca. 
Advanead  tadwiquoa  biduda  oogniUva  iBMnoiii 
aaaaaaingpartoimanoaavartima. 

Automaiad  Parformanca  Maaauramanl  lor  ToaitidUnlla: 
Indudaa  tadmotogiaa  tor  aaiaaaing  tha  paiformanca  of 

groupa  wWi  roapad  to  a  aharad  taak. 


ttol>ayrdiaiwatty.  Tha  uaa  ol  advanead 
oiaaaurinQ  dia  phydotogp  af 


avania  to 
and  to  davalap 


Mtoafeig  Tbam  Mtonbar  MmlaBon:  Automaiad  raptoeamani 
el  oiiar  loam  mambam  to  piovida  baaradion  to  a 

babiaai 


Imulalton  tor  Bwbaddad  TVabibig:  Tadmiquaa  tar  Invad¬ 
ing  dgnaliblgniaaa  at  abmddad  adiaraaitoa  into 
npataltonal  aqubmanL 


bidudaa  omk  to  medd  anvbonmanla.  uahldaa.  and 
amapena  tor  ptodonn  and  dbtitoulad  imaradlvo  abnula- 


bilalloanl  Opponanl  Mmutolton:  Uaa  ol  AKbaaad  lach- 
dquaa  to  pravldo  appiepifaia  advamaty  bahavtar  dwlng 
abmdadDn.  TMa  may  ba  alampiad  al  Mw  ptadorm  lavd. 
or  to  hbhar  tovato,  audi  aa  itovaleplng  tofaigant  auto- 
nooioua  taroaa  bi  aihatoar-laval  abrndnUon. 

NalarorUng  Tachnalogy:  Tadmotogiaa  tor  ooimaeling  aimlar 
and  hatoroganaaua  abnulaliMi  aytoama.  bidudaa 
tiavatopmato  ol  eompultotonal  ardtoaciuraa.  dtoa- 
bantffliaalon  pntooola.  wmpraaaton  and  ancryption 
tadinlquaa. 

bialruelor  OomrobManagamanl  of  Mmutolionn:  Tadi- 
niquaa  tar  htoniotar  aaup  el  abnidtotona  and  aoanaiiaa. 

daia  abaaadlon  and  rapmaantaHen  tor  aeanarto 
nwnagamoni,  biainiotor  albitoion  aaaaaamoid.  laoogni- 
tton  ef  tatoke  and  abtoagiaa. 

Mplay  Tachnolagr-  Tadmotogiaa  tor  dtaptoybig  dmdtoad 
•ovfronmaida,  bidudbig  tadmiquaa  lor  bwiaaaing  btigki- 
naato  raaekAion,  eetor  langotoalanca.  ftokM-viao, 
poamr  conaumpiton.  banapnittofy 

Haad  Meuntad  Plaplay  Taehnategy;  Aa  Mova.  but  biduding 
••'Aniquaa  tor  M),  taaar  arnlgiq  and  poamr. 

bnaga  Oanoraden:  Tadmotogiaa  tor  moddbig  and  rapraaan- 
t^  d  viaual  Imagaa.  bauaa  biduda  raaokaion.  Md- 
d-vtoar.  dtoa-raia.  data  eompraaaion  and  hlgh-rm 
dacompraaaton.  dtoa-baaa  ganaitoion.  oon«iuciion  d 
raal-aiMld  modab  from  aanaor  dtoa. 

dd  Hapite  napbya:  bidudaa  lachnologiaa  tor  audto 
'*  aound  bcalaiion.  lorca  ganartobn  to  aibnulata 
toudi. 

TtanadiwandTtachbigTadinotogy;  Davatopmani  d  dav- 
baa  and  procaaaaa  tor  aanabig  peabon  and  aMuda  d 

fdnaa'a  oyaa.  fwad.  Ingara,  handa.  ate.  and  tor  backing 
avarllma.  bauaa  biduda araigfa.  raaokaion.  aMMy. 

Voteo  Banabig  and  WaoagnMen:  Tadmotogiaa  tar  aotraeibig 
maanbig  from  apaoch  aiput.  biduiing  noik  on  phonama 
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irtMlificttion  in  ooniinuoiM 


Wghf  lnnl  boundwy 
ipMCh*  wputkMt  kidtpM^ 


Al  Md  RipMt  itedtInQWiarMtittehg  InuBatK 

ndion.  knowIndQn  wuidiQi^  knowlidQn  rapraMnUtton* 
InlaranM  wiginM,  aulotiMiid  wpiMNiDn. 

SgImiIMo  VtauidnllM:  TtchfiolDQlM  far  provMng 

or  viMMi  taMtaoM  to  oomplcfatd  dfat  and  eenotto. 
For  oxampfa,  probabay  €t  dModton  d  a  aanaor  ayaiaai 
cnn  on  ono^^aoo 

Human  Fadora  al  falaniidlaw  Olapfay  and  Oowbal;  Taeh- 
iwloBlaa  far  lapraaadlnp  dda  (a^.  tor  taabuetora),  and 
tor  pravidtoQ  aonbd  baadaoaa  to  abnidadona  and 
naOMifcad  ayatoma  and  acanartaai 

Stondardtaatton:  Conad  and  dWitoudon  d  mining  toehnot* 
ogy,  Mudtog  alandaida  tor  imaiopafabiMy  d  data,  md- 
bmadia,  abnutatton  ayaiama. 

Training  Syatoina  DaalgitfDamtepmanWupFarti  todudaa 
antogie  ptonidng  d  baining  laehndogy  ayaiama. 
aoqualion  pdiey  and  alandaida,  Imptomadalinn  and 
uaar-imiduainanl  abdaglit.  wdandad 

ptoliidiltoytooamiladmnaaa  aiudlaa.  human  and  oigan- 
iuiiond  laetoia  In  mining  taehnetogy  upiaka.  logiaiica 
andauppon.  maMananea  d  mining  qu^  ovar  dma. 
eapabitoy  upgradaa. 

Matriea  lor  Human  Mermanoa  to  Total  Syatama:  Tachno- 
lagiaa  tor  diaraeiaiiting  aHaettuanaaa  d  humm  parlor* 
manea  and  mintog  in  tarma  d  told  ayaiam  parloimanea. 
Modala  tor  rndtog-oH  tovaolmania  in  ayatam  agMbllly 
va  mining  or  aupport  tor  human  partormanoa. 
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APPENDIX  C 

THE  SIMULATION  AND  TRAINING  TECHNOLOGY  SURVEY  (STTS) 


CENTER  OF  EXCELLENCE  TRAINING  TECHNOLOGY  SURVEY 

INSTRUCTIONS 

The  attached  survey  represents  positions  held  at  the  Naval  Training 
Systems  Center  on  selected  research  topics.  These  positions  help  define  our 
research  and  development  program.  You  will  find  from  the  survey  that  our 
conceptions  are  incomplete.  We  invite  you  to  help  us  refine  and  extend  these 
conceptions  and  improve  progress  in  the  field  by  completing  the  survey.  If  you 
cannot  complete  the  entire  survey,  please  return  a  partial  response  focusing 
on  items  in  which  you  are  most  competent. 

The  survey  is  hierarchical  in  form,  as  shown  in  the  figure  on  page  2. 
That  is,  there  is  a  set  of  Military  Training  Requirements  that  require  Delivery 
Methods/Strategies  (i.e.,  approaches  for  administering  instruction),  which,  in 
turn,  require  Training  Technologies  (i.e.,  general  capabilities  required  for 
the  delivery  of  instruction) .  The  Training  Technologies  require  Enabling 
Technologies  (i.e.,  specific  capabilities  that  support  the  Training 
Technologies).  These,  in  turn,  have  milestones.  The  objective  is  to  identify 
the  importance/validity  of  each  level  in  the  hierarchy  relative  to  the  next 
higher  level.  The  procedure  is  given  in  the  following  and  an  example  of  a 
completed  survey  sheet  is  shown  in  the  table  on  page  3. 

Step  1  -  RATE  CRITICALITY  AND  COMPETENCE:  COLUMN  1 

o  Using  the  attached  Technology  Rating  Form  and  Criticality  Scale,  please 
rate  (in  column  la)  the  criticality  of  advances  in  each  of  the  Delivery 
Methods/Strategies*  shown  in  column  1  to  Military  Training  Requirements, 
exemplified  on  the  front  of  the  survey  form.  Provide  one  rating  for  each 
Delivery  Method/  Strategy  reflecting  the  need  for  progress  in  this  area  of 
military  training.  (For  example,  if  you  consider  progress  in  Embedded  Training 
"Moderately  Critical”  for  satisfying  Military  Training  Requirements  relative 
to  competing  Delivery  Methods/Strategies  of  which  you  are  aware,  place  a  "3" 
next  to  "Embedded  Training"  in  column  la) . 

o  Next,  use  the  same  form  and  the  Competence  Scale  to  rate  (in  column  lb) 
your  level  of  knowledge  or  competence  for  each  "criticality"  rating  you  have 
given  in  column  la.  (For  excunple  if  you  are  very  knowledgeable  about  Embedded 
Training,  place  a  "1"  next  to  Embedded  Training  in  column  lb). 

o  Comment  on  "Methods/Strategies"  or  "Requirements"  or  add  new  items  in 
column  5  or  any  other  place  on  the  form,  as  appropriate. 


*  Descriptions  of  selected  Delivery  Methods/Strategies  and  associated 
technologies  ar  attached. 
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INSTRUCTIONS  (CONT'D) 

o  Rate  the  criticality  and  your  competence  for  any  new  items  that  you 
contribute,  just  as  for  the  original  items. 

Step  2  -  RATE  CRITICALITY  AND  COMPETENCE:  COLUMNS  1  AND  2 
Using  the  same  procedures  as  in  Step  1: 

o  Rate  (in  column  2a}  the  criticality  of  advances  in  each  of  the  Training 
Technologies  (column  2)  to  the  corresponding  Delivery  Methods/Strategies 
( column  1 ) . 


Military  Training  Requirements 


i^~^li' 


(How  critical  are  the  Delivery  Methods /Strategies  to 
Military  Training  Requirements?) 


Delivery  Methods /Strategies 


(How  critical  are  thfe^raining  Technologies  to  the 
Delivery  tietWods/ Strategies?) 


(How  critical  are  the  Enabling  Technologies  to  the 
(Training  ’Technologies?) 


Enabling  Technologies 


(How  appropriate  jare  the  milestones  to  the 
Enabling  "technologies?) 


Milestones 


Hierarchical  Structure  of  Survey 


*  Four  Training  Technologies — Threat  Modeling  &  Common  Data  Bases,  Automated 
Scenario  Generation  &  Control,  Expert  Systems,  and  Visual/Sensor  Simulation — 
use  a  broad  development  strategy  &  are  not  associated  with  particular  Delivery 
Methods/Strategies . 
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TlCMMLOai  MIIK 


EXAMPLE  OF  A  COMPLETED  SURVEY  SHEET 
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INSTRUCTIONS  (CONT'D) 

o  Rate  (in  column  2b)  your  competence  for  each  criticality  rating. 

o  Comment  on  Training  Technologies  or  add  new  Training  Technologies  in 
column  5  or  any  other  place  on  the  form,  as  appropriate. 

o  Rate  the  criticality  and  your  competence  for  any  new  items  that  you 
contribute,  just  as  for  the  original  items. 

Step  3  -  RATE  CRITICALITY  AND  COMPETENCE:  COLUMNS  2  AND  3 

Using  the  same  procedures  as  in  Steps  1  and  2: 

o  Rate  (in  column  3a)  the  criticality  of  advances  in  each  of  the  Enabling 
Technologies  (column  3)  to  the  corresponding  Training  Technologies  (column  2). 

o  Rate  (in  column  3b)  your  competence  for  each  criticality  rating. 

o  Comment  on  Enabling  Technologies  or  add  new  Enabling  Technologies  in 
column  5  or  any  other  place  on  the  form,  as  appropriate. 

o  Rate  the  criticality  and  your  competence  for  any  new  items  that  you 
contribute,  just  as  for  the  original  items. 

Step  4  -  DEFINE  MILESTONES  FOR  ENABLING  TECHNOLOGIES 

o  Add  to  or  comment  on  the  Milestones  (column  4  of  the  Rating  Form)  for 
corresponding  Enabling  Technologies,  as  appropriate. 

Step  5  -  DESCRIBE  DELIVERY  METHODS /STRATEGIES  AND  ASSOCIATED 
TECHNOLOGIES 

o  Comment  on  the  attached  technology  descriptions,  as  appropriate.  Legible 
handwritten  annotations  made  directly  on  the  papers  are  suitable. 

o  Submit  new  or  revised  papers  to  identify  critical  research  and  development 
areas  and  issues  that  we  may  have  missed. 


NOTE:  WE  ARE  INTERESTED  IN  YOUR  RECOMMENDATIONS,  ESPECIALLY  FOR 

COOPERATIVE  EFFORTS  BETWEEN  PRIVATE  AND  PUBLIC  SECTORS, 
THAT  CAN  INCREASE  SPONSORSHIP  FOR  TRAINING  RESEARCH  AND 
IMPROVE  THE  QUALITY  OF  RESEARCH  PRODUCTS. 
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TECHNOLOGY  RATING  FORM 


NAME _ 

organization _ 

posmoN/mLE _ 

EDUCATIONAL  BACKGROUND 
YEARS  EXPERIENCE  IN  FIELD_ 
MAIUNO  ADDRESS _ 


PHONE  NO. 


CRITICALITY  SCALE 

4  5 

Minimillv  Critic*] 
Progrctt  in  capability 
contnbutea  liole  to 
none. 


COMPETENCE  SCALE 


1  2 

3  4 

5 

Very  KnowledEeable/ 

Knowledseable.  Have 

Limited  Knowledsc.  Have 

Comoetent.  Have 

some  expeiience  in  this 

litUe  knowledge  id  this 

wotted  extensively  in 

area.  Am  familiar  with 

area. 

this  area.  Am 

current  work. 

competent  in  this 

subject. 

I 

Very  Critical 
Pragreu  u  more 
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The  following  topics  are  included  to  stisulats  interest  in  defining 
additional  eethode  and  technologiee.  Pleaee  contribute  to  these  topics  and 
identify  additional  Itaaa,  ee  appropriate. 
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EMBEDDED  TRAINING 


DESCRIPTION  OF  TECHNOLOGY 

The  Chief  of  Naval  Education  and  Training,  Embedded  Training  Task  Force 
(1985,  Nov.)»  defined  embedded  training  as  "training  that  is  provided  by 
capabilities  built  into  or  added  onto  operational  systems,  subsystems,  or 
equipment,  to  enhance  and  maintain  the  skill  proficiency  of  fleet  personnel" 
(para  4.1).  Embedded  training  offers  a  means  of  providing  initial, 
advanced,  refresher,  predeployment,  and  mission  rehearsal  training  at 
deployed  or  remote  sites. 

Embedded  training  can  be  configured  in  various  ways  to  meet  specific 
applications.  Embedded  training  can  be  fully  embedded  in  the  operational 
equipment,  it  can  be  provided  by  adding  on  external  equipment  to  the 
operational  equipment  (e.g.,  target  generators),  or  operational  equipment 
can  be  connected  to  training  equipment  with  umbilical  cables.  Whichever 
form  it  takes,  embedded  training  usually  has  the  following  two 
characteristics : 

a.  It  uses  operational  equipment  (e.g.,  consoles,  displays,  indicators) 
as  the  primary  training  media.  Trainee  interaction  with  the  operational 
equipment  is  required.  Operational  data,  signals,  or  targets  are  simulated 
or  stimulated  on  the  operational  ecjuiixnent  for  operator  training.  Real 
world  faults,  malfunctions,  and  interferences  are  simulated  or  stimulated  on 
the  operational  equipment  for  maintenance  training. 

b.  It  should  include  instructional  support  features  to  manage  the 
training  process  since  instructors  are  often  unavailable.  Automated 
development  and  control  of  training  scenarios,  control  of  lesson  content, 
and  adjustment  of  lesson  difficulty  may  be  required.  A  mechanism  for 
trainee  performance  measurement  and  feedback  is  required.  Records 
management  and  exercise  storage  functions  are  often  included. 

In  the  past,  embedded  training  has  often  used  engineering  test 
approaches  to  inject  signals  into  operational  equipment.  These  approaches 
allow  for  practice,  but  provide  little  training.  Only  when  instructional 
technologies  are  included  does  an  embedded  training  system  exist.  Many  of 
the  instructional  tasks  conducted  by  instructors  must  be  automated  in 
embedded  training  systems  because  instructors  are  not  available  or  manning 
is  less  than  at  shore-based  facilities.  Instructional  technologies  may  be 
classified  into  two  main  categories:  those  with  general  applicability  to 
all  forms  of  embedded  training,  and  those  with  special  emphasis  on  tactical 
and/or  team  training. 

GENERAL  EMBEDDED  TRAINING  INSTRUCTIONAL  TECHNOLOGIES 

The  following  instructional  technologies  are  applicable  in  all  forma  of 
embedded  training: 

a.  Automated  performance  measurement  and  expository  feedback:  monitor 
the  performance  of  the  trainee  and  provide  information  about  errors  and 
corrective  actions.  If  the  trainee  is  not  provided  feedback  on  the  quality 
of  his  or  her  performance,  little  learning  can  be  expected. 

b.  Automated  adaptive  instruction:  continuously  assesses  trainee 
strengths  and  weaknesses  and  adjusts  the  course  of  instruction  to  build  on 
strengths  and  focus  on  weaknesses. 
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TACTICAL/TEAM  EMBEDDED  TRAINING  INSTRUCTIONAL  TECHNOLOGIES 

The  following  technologies  are  specific  to  embedded  training  for  tactics 
and  team  applications: 

a.  Intelligent  platforms:  make  decisions  (independent  of  an 
instructor)  to  realistically  simulate  the  maneuvers  and  actions  of  both 
friendly  assets  and  targets  during  a  dynamic  training  exercise. 

b.  Missing  team  member  simulation:  removes  the  necessity  to  access 
whole  teams  or  subteams  when  the  desire  is  to  training  single  te£un  members 
or  subteeuns.  Although  virtually  all  training  involves  team  interactions, 
all  members  of  all  teams  are  not  always  available  for  training.  Simulating 
missing  team  members  allows  training  to  proceed  and  also  allows  control  of 
the  level  of  expertise  of  the  missing  team  members. 

TRAINING  MANAGEMENT  AND  ORGANIZATIONAL  ISSUES 

Training  development,  coordination,  and  management  are  important  issues 
in  embedded  training  due  to  the  reduced  manning  levels  of  instructors  and 
course  administrators,  and  the  diverse  locations  or  embedded  training. 

These  technologies  fall  into  three  main  categories: 

a.  Enhanced  lesson  and  scenario  authoring:  provide  the  capabilities 
for  supervisors  or  other  subject  matter  experts,  who  are  not  training 
experts,  to  build  and  modify  training  scenarios  that  will  provide  realistic, 
mission-relevant  lessons  that  include  the  instructional  capabilities 
discussed  above. 

b.  Automated  record  management:  maintains  training  records  of  all 
trainees  and  makes  these  records  available  when  needed  to  evaluate 
additional  training  requirements,  promotion  potential,  reassignment 
potential,  etc. 

c.  Coordination  of  embedded  training:  refers  to  the  organizational 

issues  of  standardizing  and  validating  the  embedded  training  that  is  given 
at  various  sites.  Without  such  coordination,  it  is  not  possible  to  ensure 
that:  (1)  individuals  trained  at  one  site  will  receive  the  same  level  of 

training  that  was  provided  at  other  sites;  (2)  individuals  will  be  able  to 
continue  their  training  path  at  alternative  sites;  or  3)  that  training 
deficiencies  can  be  identified  and  corrected  at  future  sites. 

ENGINEERING  TECHNOLOGY  ISSUES 

Even  though  many  of  the  technologies  to  stimulate  operational  equipment 
with  signals,  targets,  simulated  malfunctions,  and  other  functions  necessary 
for  embedded  training  already  exist,  technologies  to  maximize  the  efficiency 
and  quality  of  stimulation  are  required.  Some  of  the  engineering  technology 
issues  include: 

a.  How  to  determine  at  which  point  in  the  operational  system  (e.g.,  at 
the  display,  at  the  main  bus,  at  the  radar  dish,  etc.)  signals  should  be 
injected  or  the  system  otherwise  stimulated. 

b.  How  to  ensure  that  the  operational  system  can  be  brought  out  of 
training  mode  and  back  to  operational  mode  (juickly  and  efficiently. 

c.  How  to  ensure  that  embedded  training  scenarios  do  not  damage  or 
otherwise  degrade  the  operational  system. 
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d.  How  to  ensure  weapons  safety  during  training. 

e.  How  to  ensure  the  safety  of  the  ship  and  crew  during  training. 

f.  How  to  choose  between  varieties  of  embedded  training  (e.g.,  fully 
embedded,  strap-on,  umbilical,  etc.). 


CRITICAL  TECHNOLOGY  CHALLENGES  IN  EMBEDDED  TRAINING 
o  Instructional  Technologies 

-  General  Technologies 

—  Automated  Performance  Measurement 
and  Expository  Feedback 
—  Automated  Adaptive  Instruction 

-  Technologies  Specific  to  Tactical/Team  Training 
—  Intelligent  Platforms 

—  Missing  Team  member  Simulation 

o  Training  Management  and  Organizational  Technologies 

-  Enhanced  Lesson  &  Scenario  Authoring 

-  Automated  Record  Management 

-  Coordination/Standardization  of  Embedded  Training 
o  Embedded  Training  System  Engineering  Technologies 

-  Stimulation  Technologies 

-  Conversion  from  Training  to  Operation  Modes 

-  System  Fail  Safe  Technologies 

-  Technologies  to  Ensure  Safety  of  Weapons  and  Personnel 

-  Methods  to  choose  Between  Types  of  Embedded  Training 
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MILESTONES 


Instructional  Technologies 


TECHNICAL  AREA 

BY  1995 

BY  2000 

Automated  Performance 
Measurement  & 

Expository  Feedback 

Develop  guidelines  for 
implementing 

Implement  in  new  or 
fielded  systems 

Automated  Adaptive 
Instruction 

Develop  guidelines  for 
cognitively  structuring 
&  adaptively  sequencing 
instruction  (with 
algorithms  to  be 
followed) 

Implement  in  new  or 
fielded  systems 

Intelligent  Platforms 

Develop  IP  requirements 
for  all  warfare  areas 

Develop  authoring 
system(s)  for  IPs 

Missing  Team  Member 

Sim. 

Develop  methods  to 
implement  for  auditory 
feedback 

Develop  general  methods 
to  implement  for  other 
than  auditory 

Training  Management  &  Organizational  Technologies 


TECHNICAL  AREA 

BY  1995 

BY  2000 

Enhanced  Lesson  & 
Scenario  Authoring 

Develop  prototype  tool 
for  automated  scenario 
authoring 

Develop  a  general  tool 
for  automated  scenario 
authoring 

Automated  Record 
Management 

Develop  a  freunework  & 
plan  for  Navy-wide 
automated  ET  record 
management  system 
(possibly  modeled  on 

Air  Force's) 

Implement  a  Navy-wide 
automated  ET  record 
management  system 

Coordination  & 
Standardization 

Identify  &  investigate 
factors  necessary  to 
coordinate  & 
standardize 

Develop  guidelines  to 
coordinate  & 
standardize 
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Engineering  lechnologies 


TECHNICAL  AREA 

BY  1995 

BY  2000 

System  Stimulation 

Identify  issues 
critical  to  stimulating 
actual  equipment  (AE) 

Implement  Navy-wide 
guidelines  for 
stimulating  AE 

System  Mode  Conversion 

Identify  critical 
factors  for  conversion 
from  ET  to  operational 
modes  for  selected 
weapon  systems 

Implement  guidelines 
for  conversion  from  ET 
to  operation-  al  modes 
for  all  weapon  systems 

System  Fail  Safe 

Identify  critical 
factors  for  maintaining 
fail  safe  for  selected 

ET  applications 

Implement  guidelines 
for  fail  safe  for  all 

ET  applications 

System  Safety  Issues 

Identify  critical 
factors  to  ensure 
safety  of  weapon 
systems  and  personnel 
during  ET 

Implement  guidelines 
for  safe  ET 
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DESCRIPTION  OF  TECHNOLOGY 

Networking  when  used  in  a  training  context  encompasses  the 
interconnection  of  training  systems  and  operational  equipment.  Considerable 
success  has  been  achieved  by  the  Army  in  networking  combat  simulators  at 
different  geographical  locations.  For  example,  the  Army  can  effectively 
conduct  combined  arms  training  using  tank  simulators  at  Ft.  Knox,  KY, 
networked  to  helicopter  simulators  at  Ft.  Rucker,  AL,  and  Bradley  Fighting 
Vehicles  at  Ft.  Banning,  GA.  This  technology  has  come  to  be  known  as  SIMNET 
(simulator  networking).  To  date  the  Navy  and  Air  Force  have  made  very 
limited  use  of  simulator  networking. 

Simulator  networking  holds  great  potential  for  enhancing  tactical  team 
training  from  small  units  up  to  joint  service  exercises.  Networking  is  the 
cornerstone  technology  for  the  Tactical  Combat  Training  System  (TCTS) .  This 
battle  group  trainer  will  require  the  networking  of  simulations  running  on¬ 
board  operational  platforms  at  sea  and  in  the  air  with  land  based 
simulations.  Simulator  networking  has  the  potential  to  allow  the  Navy  to 
realize  more  benefits  from  its  existing  inventory  of  training  systems. 

Many  technical  issues  must  be  resolved  before  heterogeneous  trainers  can 
be  productively  used  in  a  network.  The  major  issues  in  interconnecting 
trainers  are: 

Environmental  Parameters.  The  simulated  environments  in  the  various 
host  computers  must  correlate.  There  is  not  even  a  standard  definition  of 
the  environmental  parameters,  much  less  agreement  in  their  values. 

Data  Bases  and  Mathematical  Models.  The  environment, vehicles  and 
weapons  systems  in  a  simulation  are  represented  by  mathematical  models. 

There  is  no  standard  for  either  the  mathematical  models  or  the  data  bases 
for  the  various  entities. 

Data  Rate  Limitations.  The  amount  of  data  that  can  be  interchanged 
between  interconnected  simulations  is  limited.  Transmission  of  large 
amounts  of  data  is  not  only  costly  in  terms  of  equipment,  but  leads  to 
unacceptable  delay  in  the  availability  of  the  data. 

Network  Security.  Much  of  the  data  transmitted  between  trainers  in  a 
tactical  training  exercise  is  classified.  A  secure  data  network  must  be 
provided. 

The  issues  associated  with  these  problems  are  the  following: 

o  How  to  ensure  that  the  simulated  environments  in  two  host  computers 
correlate? 

o  How  well  must  the  environments  correlate  to  produce  adequate  training? 
How  to  handle  differences  in  level  of  detail  between  two  models? 

o  How  to  communicate  changes  in  environment  in  one  host  computer  to 
other  host  computers? 

o  How  to  provide  low  cost  encryption  and  decrypting  devices  that  have 
minimal  propagation  delays? 
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milestones 


TECHNICAL  AREA 

BY  1995 

BY  2000 

Environmental 

Parameters 

o  Define  critical 
environmental  parameters 
o  Determine  degree  of 
correlation  required  for 
adequate  training 

o  Develop  environmental 
models  and  improve 

o 

Distributed  Interactive 
Simulation  (DIS)  Standard 

Data  Bases/ 
Mathematical 

Models 

o  Define  critical  data 
base  entities  and  math 
model  parameters 

o  Develop  methods  and 
techniques  for  handling 
different  levels  of  detail 

o  Develop  reusable  models 
and  improve  DIS  standard 

Data  Rate 
Limitations 

o  Determine  bandwidth 
requirements  for  networked 
trainers 

o  Develop  data  compression 
techniques 

o  Improve  DIS  standard  and 
use  techniques  in  existing 
training  systems 

Network  Security 

o  Develop  low  cost  crypto 
devices 

o  Use  devices  in  networked 
training  applications 
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VIRTUAL  ENVIRONMENT  TRAINING  TECHNOLOGY  (VETT) 


DESCRIPTION  OF  TECHNOLOGY 

Virtual  Environment  (VE)  technologies  are  being  developed  to  provide 
more  efficient  and  effective  interfaces  between  humans  and  computers.  The 
motivating  premise  for  developing  VE  technologies  is  that  existing  user- 
computer  interfaces  are  unnatural  and  inefficient.  Terminal  displays  and 
keyboards  were  adequate  for  users  as  long  as  the  other  side  of  the 
interface,  the  computer,  was  slow  and/or  non-inter active.  The  terminal- 
keyboard  interface  has  limited  capability  in  both  directions.  Almost  all 
the  information  flowing  across  the  user-computer  interface  is  one¬ 
dimensional  and  symbolic  in  the  form  of  a  streaun  of  alphanumeric  characters. 
The  terminal  stimulates  a  small  part  of  only  one  of  the  user's  sensory 
systems  (vision)  and  the  keyboard  is  stimulated  by  a  relatively  limited  set 
of  user  mechanical  actions  (typing).  VE  technology  has  the  goal  of 
developing  natural,  multi-modality,  user-transparent,  human-computer 
interfaces  by  providing  the  user  with  displays  for  all  of  his  sensory 
systems  and  transducers  which  can  interpret  the  user's  control  actions. 

Communication  in  the  natural  world  between  a  man  and  his  environment  is 
rich  and  complex.  All  of  his  senses  are  stimulated  and  there  is  wide  range 
of  possible  actions  he  can  use  to  affect  his  environment.  The  existing 
user-computer  interface  (in  most  cases)  results  in  the  user  perception  of 
observing  through  a  window  as  opposed  to  being  a  participant  in  an  3- 
dimensional,  interactive,  totally  experienced  environment. 

The  potential  of  VE  has  been  recognized  by  the  popular  press  in  feature 
articles  describing  "virtual  reality,"  "virtual  environments,"  "artificial 
reality,"  "cybernetics,"  etc.  In  general,  the  articles  miss  the  point. 

They  usually  emphasize  the  tools  used  to  enable  VE's  and  exotic  applications 
of  the  concept.  The  point  missed  is  that  VE  is  a  communications  medium. 

The  specific  hardware  and  software  tools  used  by  the  medium  and  the  specific 
applications  of  the  medium  are  not  essential  to  the  concept.  VE  is  a  medium 
of  communication  just  as  books,  telephones,  televisions,  etc.  are 
communication  media. 

VE  VS.  SIMULATION  VS.  TELEPRESENCE 

How  does  VE  technology  differ  from  man-in-the-loop  simulation  and 
telepresence?  The  answer  is  that  the  differences  are,  or  at  least  should 
be,  transparent  to  the  user.  Ideally,  the  user  side  of  the  interface  is  the 
same  in  all  three  technology  areas.  The  interface  is  such  that  the  user 
perceives  himself  to  be  immersed  in  an  environment.  The  differences  between 
the  three  related  technologies  are  on  the  machine  or  computer  side  of  the 
interface.  In  the  case  of  telepresence,  the  environment  is  created  by  a 
remotely  located  machine  sensor  system,  e.g.  a  remotely  piloted  vehicle, 
which  is  operating  in  the  real  world.  In  the  case  of  man-in-the-loop 
simulation,  the  environment  is  a  mix  of  computer  generated  and  physical 
representations  of  a  real  world  operator  environment.  In  the  case  of  VE 
technology,  the  environment  is  totally  (ultimately)  synthesized  by  computer. 
The  differences  between  training  simulators  and  VE  are  subtle.  VE 
technology  is  designed  to  emphasize  the  user  or  human  side  of  the  interface 
and,  ideally,  is  independent  of  the  machine  side  of  the  interface,  e.g.,  a 
VE  system  for  pilot  training  might  use  the  same  interface  hardware  set  as  a 
VE  system  for  sonar  operator  training.  Simulation  technology  emphasizes  the 
machine  side  of  the  interface,  e.g.,  simulators  designed  for  F-14  pilot 
training  are  not  useable  for  P-3  pilot  training.  The  emphasis  of  VE 
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technology  is  high  fidelity  stimulation  of  the  user's  sensory  systems.  The 
emphasis  of  simulation  technology  is  high  fidelity  modeling  of  real  world, 
operational  hardware.  Another  distinction  is  that  VE's  need  not  have  a  real 
world  counterpart.  The  VE  need  not  be  constrained  by  laws  of  physics,  for 
example.  Whereas  simulated  environments  generally  strive  to  have  high 
fidelity  to  the  real  world  environment  being  simulated.  Man-in-the-loop 
simulators  are  a  sub-class  of  VE's  in  that  some  simulator  system  components, 
such  as  the  visual  display  of  the  extra-cockpit  environment  in  a  flight 
simulator,  can  be  considered  to  be  non-platform  specific  while  other  system 
components,  such  as  flight  controls  are  highly  platform  specific. 

Another  premise  driving  the  development  of  VE  technology  is  that 
symbolic  interfaces  (communication  by  reading  and  writing)  is  in  itself 
inefficient  and  unnatural.  The  history  of  communication  didn't  end  with  the 
invention  of  the  printing  press.  There  is  no  natural  law  that  says  that 
communication  media  must  use  alpha-numeric  symbols.  Communication  which 
involves  all  of  the  senses  to  the  limits  of  their  capabilities  is  both  more 
efficient  and  more  effective.  When  communication  is  interactive,  the 
efficiency  and  effectiveness  improves  still  further.  Natural,  direct,  total 
sensory  experiences  are  more  effective  than  translating  and  processing 
alpha-numeric  symbols. 

VE's  have  been  and  will  continue  to  be  developed  under  many  different 
names.  The  name  is  just  a  concept  for  unifying  and  giving  direction  to 
diverse  developments.  Some  examples  of  how  developments  are  progressing 
toward  VE's  (without  being  called  VE's)  are: 

a.  Literacy  -  As  media  which  provide  more  sensory  stimulation  become 
more  and  more  available  there  is  less  and  less  motivation  for  people  to 
become  proficient  alpha-numeric  symbol  manipulators.  Primary  school 
students  spend  and  enjoy  their  time  with  television  and  video  games  much 
more  than  newspapers  or  books.  The  television  and  video  game  media  are 
closer  to  natural  experiences  than  text.  The  recently  reported  multi-year 
decline  in  SAT  verbal  scores  may  be  symptomatic  of  the  trend. 

b.  Documentation  -  Historic  documentation  in  textbooks  has  given  way, 
first  to  photographs,  then  to  motion  pictures  and  most  recently  to  video 
tapes.  Each  stage  representing  a  step  toward  more  natural  experiences.  The 
domination  of  home  video  Ccuneras  over  motion  picture  cameras  (despite  a  loss 
in  image  fidelity)  attests  to  the  desirability  of  immediate  feedback  and 
user  interaction. 

c.  Desktop  Computer  Interfaces  -  The  recent  surge  in  popularity  of 
graphic  user  interfaces  (GUI),  mouse,  joystick,  trackball,  touch  screens, 
etc.,  demonstrate  the  advantages  of  such  devices  over  alpha-numeric  terminal 
and  keyboard  interfaces. 

d.  Visual  Simulation  Technology  -  Man-in-the-loop  simulators  have  been 
going  down  the  road  toward  VE's  for  decades.  The  appearance  of  the  real 
world  has  progressed  from  3-dimensional  scale  physical  models  (terrain 
boards)  or  photographs  to  real  time  computer  image  generation  (CIG).  The 
loss  in  image  fidelity  has  been  greatly  outweighed  by  the  increase  in  real 
time  interaction  and  flexibility  in  training  scenarios  allowed  by  CIG. 
Simulation  of  operational  cockpit  displays  is  progressing  from  the  use  of 
synthetically  stimulated  actual  instruments  to  simulation  of  the  appearance 
of  cockpit  instruments  using  graphics  on  a  CRT  face. 

e.  Interpersonal  Communication  -  The  last  major  breakthrough  in 
interpersonal  communication  was  the  telephone.  But,  video  teleconferencing 
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is  considered  to  have  great  potential  and  technical  obstacles  are  being 
overcome . 

VE  TECHNOLOGY  STATUS 

The  capabilities  of  a  user  to  absorb,  process  and  transmit  information 
are  large  but  finite  quantities.  The  development  of  affordable,  high 
capacity  computer  systems  which  can  match  users'  information  flow  rates  is 
happening  now.  The  time  for  developing  VE  systems  is  ripe. 

VE  technology  can  be  divided  into  interface  technologies  and  environment 
generating  technologies.  Environment  generating  technologies  overlap  to  a 
high  degree  with  simulation  technologies  and  will  not  be  discussed  in  any 
detail  in  this  document.  The  interface  technologies  associated  with  VE's 
are  those  which  have  received  the  most  attention  in  the  popular  literature. 
The  interface  technologies  include  those  which  stimulate  the  user ' s  sensory 
system  (displays,  in  the  generic  sense)  and  those  which  measure  the  user's 
responses  (transducers). 

DISPLAY  SYSTEMS  -  Displays  include  visual,  auditory,  haptic  (cutaneous  and 
kinesthetic)  and  other  (smell,  taste,  etc.).  Ideally  a  perfect  VE  display 
would  provide  sensory  stimuli  indistinguishable  from  those  provided  by  real 
world  counterparts  of  the  VE. 

a.  Visual  Displays  -  The  visual  displays  usually  associated  with  VE's 
are  head  mounted  displays  (HMD).  This  isn't  an  absolute  requirement.  It 
just  so  happens  that  helmet  mounted  displays  are  capable  of  providing  the 
unlimited  total  field  of  view,  which  compels  a  total  immersion  experience, 
at  a  much  lower  coat  than  the  multiple  projector,  full  field  of  view  dome 
visual  systems  such  as  those  used  in  flight  training  simulators.  The  image 
quality  of  the  consumer  affordable  HMD  s  is  now  somewhat  deficient  when 
compared  to  that  available  in  a  simulator  dome  visual  system,  but  it 
suffices  (for  most  observer's)  to  provide  the  "suspension  of  disbelief" 
necessary  for  an  immersive  experience.  Commercial  head  mounted  display 
systems  are  available  off-the-shelf  at  costs  ranging  from  a  few  hundred 
dollars  to  $15,000.  More  sophisticated  head  mounted  display  systems  have 
been  developed  for  military  operational  and  training  simulator  applications 
at  costs  ranging  from  $50K  to  $lM-t-.  The  primary  performance  differences 
between  the  low  cost  commercial  systems  and  the  high  cost  military  systems 
are:  military  systems  are  see-through  (the  computer  generated  imagery 
overlays  the  non-virtual,  visible,  external  environment)  while  the 
commercial  systems  are  generally  opaque  to  the  outside  world  (all  visually 
sensed  information  is  provided  by  the  VE);  and  the  military  systems 
generally  have  higher  resolution  but  smaller  instantaneous  fields  of  view. 
The  major  problem  associated  with  HMD's  is  instability  of  the  displayed 
image  caused  by  the  time  lag  between  a  head  movement  and  the  display  of  the 
correct  imagery  for  the  new  head  position. 

b.  Audio  Displays  -  High  fidelity,  head  mounted  stereo  systems  are 
commercially  available  and  are  being  used  in  laboratories  for  VE  system 
development.  The  physical  differences  between  users'  hearing  sensor  systems 
results  in  a  requirement  for  individual  calibration  if  the  sound 
localization  capability  of  the  user  is  to  be  effectively  stimulated. 

c.  Haptic  Displays  -  Providing  effective  stimulation  of  the  cutaneous 
sensor  system  is  a  relatively  new  area  with  lots  of  room  for  improvement. 
Systems  which  provide  forces  to  fingers,  hands  and  arms  have  been  developed 
with  some  success.  Cutaneous  sensor  stimulation  systems  have  been  used  in 
simulators  in  the  form  of  g-seats  and  g-suits.  Stimulation  of  the 
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kinesthetic  senses,  which  allow  the  user  to  sense  his  own  limb  position  and 
acceleration  is  a  wide  open  area  for  further  investigation.  Again,  training 
simulators  have  used  motion  platforms  to  provide  onset  acceleration  cues  for 
some  time. 

d.  Other  Displays  -  Stimulating  smell,  taste  and  other  senses  is  still 
at  the  concept  stage. 

TRANSDUCER  SYSTEMS  -  Transducers  include  those  required  for  measuring 
position  and  attitude  of  the  user's  eyes,  head,  fingers,  hands,  arms  and 
other  body  parts.  Other  transducers  include  those  required  for  voice  and, 
eventually,  brain  or  other  nervous  system  electrical  emanations/properties. 

a.  Eye  Tracking  Systems  -  An  ideal  VE  visual  display  system  would  take 
maximum  advantage  of  the  limitations  of  the  human  visual  system.  The  human 
visual  system  has  its  highest  resolution  capability  in  its  central  field. 
This  capability  falls  off  rapidly  for  the  peripheral  field.  An  efficient  VE 
visual  display  should  take  advantage  of  this  situation  by  providing  high 
resolution  imagery  to  the  eye's  central  field  and  degrade  to  lower 
resolution  at  large  off-axis  angles.  Therefore,  in  an  ideal  system,  the 
look  direction  of  the  eye  must  be  measured.  Eye  tracking  technology  has 
been  under  development  for  many  years  and  there  are  commercial  systems 
available  off-the-shelf  suitable  for  experimentation  and  laboratory  systems. 
However,  there  are  no  systems  available  which  are  reliable,  affordable  and 
robust  enough  for  use  in  a  consumer  product. 

b.  Head  Tracking  Systems  -  Head  tracking  systems  have  been  under 
development  for  decades.  The  primary  application  has  been  helmet  mounted 
sights  and  HMD  stabilization  in  military  weapon  systems.  More  recently  the 
technology  has  been  used  in  simulator  and  VE  applications.  Systems  are 
available  off-the-shelf.  The  performance  of  such  systems  is  deficient  in 
several  areas  and  better  head  tracking  systems  are  still  recniired. 

c.  Hand,  Finger  Tracking  Systems  -  There  are  systems  available  off-the- 
shelf  at  costs  ranging  from  $50  to  $50,000.  Their  capabilities  range  from 
2-dimensional  hand  position  (such  as  a  mouse  or  joystick)  to  finger  position 
sensors  which  measure  up  to  20  variables. 

d.  Voice  Sensing/Recognition  Systems  -  A  speaker  independent, 
continuous  speech  recognition  system  is  a  goal  which  hasn't  been  achieved. 

As  with  many  of  the  other  components  of  VE  technology,  significant 
development  will  be  required  before  a  user  transparent  voice  interface  is 
available  with  the  qualities  of  affordability  and  reliability.  The  current 
state  of  the  art  requires  individual  calibration  for  anything  other  than  a 
few  discrete  words. 

e.  Other  Sensing  Systems  -  There  are  several  other  characteristics  of 
the  user  which  can  be  sensed.  However,  most  of  them  are  involuntary  and 
their  utilization  in  a  VE  system  would  be  purely  speculative. 

VE  APPLICATIONS 

VE  technology  is  a  concept.  It  is  a  communications  medium.  It  is  a 
tool  which  facilitates  the  man-machine  interface.  As  a  medium  it  can  be 
applied  to  a  variety  of  applications  just  as  print,  the  telephone, 
photography,  television,  etc.,  have  an  unlimited  number  of  applications. 

The  application  developer's  problem  is  to  determine  what  applications  will 
be  accomplished  more  efficiently,  effectively,  and  affordably  with  this 
tool.  But,  since  the  tool  has  the  potential  for  a  wide  variety  of 
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applications,  it  need  not  be  the  best  tool  for  each,  or  for  that  matter,  any 
of  the  applications  to  justify  its  existence.  Consider  a  desktop  computer 
system.  The  desktop  is  a  single  system  which  performs  a  variety  of 
functions.  It  could  be  argued  that  each  function  could  be  performed  more 
efficiently  by  a  dedicated  system  e.g.,  word  processor,  filing  system, 
graphics  generation,  computer  terminal,  interoffice  communication,  facsimile 
transmission,  etc.  But  no  collection  of  dedicated  systems  is  more  efficient 
for  accomplishing  all  of  the  applications. 

Speculation  on  the  potential  applications  of  VE  technology  together  with 
some  thoughts  on  a  future  society  based  on  VE's  are  listed  below: 

a.  VEHICLE  OPERATOR  TRAINING  -  A  single  VE  interface  could  provide  the 
training  environment  for  any  type  of  vehicle  from  a  car  to  a  high 
performance  aircraft.  The  more  complex  the  system  the  more  likely  this 
application  is  viable.  A  fly-by-wire  aircraft  with  multi-function  graphics 
and  head-mounted  display  environment  is  conceptually  easier  to  "virtualize" 
than  a  bicycle  environment  (given  the  current  state-of-the-art  of  haptic 
displays) . 

b.  TELEOPERATION/TELEPRESENCE  -  VE  technology  can  be  used  to  provide  a 
more  efficient  and  effective  interactive  interface  between  a  remote  operator 
and  the  task  being  performed.  The  VE  is  performing  well  when  the  user 
doesn't  know  it's  there. 

c.  VIRTUAL  OFFICE  -  All  operations  associated  with  business  office 
operation  could  be  carried  out  in  a  VE  including  teleconferencing, 
telephoning,  desktop  computer  system  operation,  filing,  etc.  Eventually,  if 
VE's  become  as  common  and  affordable  as  the  telephone  is  today,  there  would 
be  dramatic  reduction  in  physical  commuter  travel  and  business  trips. 

d.  RETAIL  BUSINESS  -  A  VE  interface  to  a  wide  area  network  would  enable 
virtual  shopping,  complete  with  interactive  product  demonstrations,  trying 
on  clothes,  etc.  Virtual  malls.  Virtual  car  showrooms  and  test  drives. 
Virtual  interior  decorating. 

e.  COMPUTER  AIDED  DESIGN  -VE's  for  design  of  just  about  anything  are 
possible.  Coupled  with  advanced  computer  systems  to  do  all  of  the 
calculations  and  insure  obedience  to  the  laws  of  physics  anyone  would  have 
the  capability  to  design  anything.  But,  just  as  a  camera  doesn't  make 
everyone  a  photographer,  designers  will  still  need  education  and  skills. 

f.  BUILDING  AND  CONSTRUCTION  -  Virtual  landscaping.  Virtual  3-D  plans 
and  blueprints.  Virtual  site  location/new  construction  impact. 

g.  TELESOCIALIZATION  -  Extend  the  current  concept  of  forums  on  computer 
networks  with  the  increased  sensory  stimulation  possible  with  VE  technology 
and  a  virtual  "common"  is  a  possibility.  The  virtual  common  allows  multiple 
participants  in  a  never  ending  interactive  discussion  which  can  be  joined  or 
left  at  any  time.  Each  participant  can  be  seen  and  heard  by  the  others. 

h.  RECREATION  &  ENTERTAINMENT  -  Virtual  theater.  Virtual  (vicarious) 
participation  in  theater,  sports,  fictitious  situations.  "Feelies."  Virtual 
vacations. 

i.  MEDICAL  -  VE's  for  paralyzed,  blind,  crippled,  learning  disabled, 
etc.  to  lessen  effects  of  their  handicaps.  Virtual  surgery  for  practice, 
experimentation,  pre-operation  briefing. 
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j.  EDUCATION  -  Virtual  computer  aided  instruction.  Virtual 
encyclopedia . 

k.  SCIBNCE/RESEARCH  -  VE's  allow  visualization  and  hands  on 
manipulation  and  study  of  all  realms  of  investigation  from  sub-atomic 
particles  to  galaxies. 

l.  CONNECTIONS  -  The  VE  equivalent  to  the  cellular  telephone  will  allow 
all  of  the  VE  applications  to  be  realized  in  a  man-portable  implementation. 

m.  QUESTIONS  -  As  implied  by  the  speculations  listed  above,  a  future 
society  based  on  the  a  VE  medium  will  have  many  far-reaching  impacts  on  all 
phases  of  society.  What  will  wealth  mean  when  the  accouterments  associated 
with  today's  wealth  are  available  to  be  experienced  in  a  VE?  What  will  the 
impact  be  on  the  travel  industry  when  there  is  no  need  to  travel  to 
experience  a  different  location?  What  are  the  impacts  on  energy 
conservation  and  environment  protection? 


CRITICAL  TECHNOLOGY  CHALLENGES  IN  VE 

VETT  OBJECTIVES  -  The  Naval  Training  Systems  Center  has  started  a  VE 
Training  Technology  (VETT)  project  to  develop  and  evaluate  the  application 
of  VE  technology  to  military  training  systems.  Although  the  potential  is 
obvious,  the  benefits  in  terms  of  cost  and  training  effectiveness  must  be 
analyzed  and  demonstrated  prior  to  the  introduction  of  the  medium  in 
military  training  syllabi.  Questions  to  be  addressed  by  this  exploratory 
development  project  include:  What  types  of  training  can  be  improved,  in 
terms  of  training  cost  and/or  skill  acquisition  and  retention  using  VE's? 

How  is  an  instructional  system  designed  and  developed  using  the  VE  medium? 
What  technical  performance  characteristics  are  required  in  a  VE  system  for 
specific  training  applications?  What  are  the  existing  VE  deficiencies  that 
must  be  overcome  to  allow  utilization  in  specific  training  applications? 

VETT  APPROACH  -  The  VETT  project  includes  the  following  tasks: 

a.  Development  of  a  VE  laboratory.  This  includes  a  survey  of  available 
VE  system  components  and  assessment  of  suitability  of  the  components  for  the 
military  training  application.  Also  included  are  visits  to  existing  VE 
laboratories  for  discussions  and  demonstrations  which  will  assist  in  the 
laboratory  development  process. 

b.  Analysis  of  existing  military  training  courses  to  identify  and 
prioritize  candidates  for  improvement  through  application  of  VE  technology. 

c.  Demonstrate  and  evaluate  existing  VE  technology  to  determine  whether 
the  specific  training  task  candidates  can  be  effectively  performed  in  the  VE 
by  someone  already  skilled  in  the  task. 

d.  Develop/modify/enhance  virtual  technology  components  as  required  to 
allow  task  performance  for  training  task  candidates. 

e.  Develop  the  instructional  system  design  for  the  candidate(s)  chosen 
for  training  effectiveness  evaluation  making  maximum  use  of  VE  technology. 

f.  Perform  training  effectiveness  evaluation(s)  through  comparison  of 
students  trained  with  conventional  techniques  using  the  same  performance 
criteria. 
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g.  For  thct»e  candidate  VE  training  tasks  which  are  demonstrated  to  be 
equally  or  more  effective  than  conventional  training,  evaluate  the  cost 
effectiveness. 

h.  Transition  candidate  training  systems  to  field  evaluation. 

i.  As  VE  technology  improves  and  experience  in  application  of  the 
technology  to  military  training  systems  grows,  iterate  the  above  process  and 
extend  to  additional  training  areas  while  improving  implementation  on 
previously  investigated  training  areas. 


MILESTONES 


Training  Technologies 


TECHNICAL  AREA 

BY  1995 

BY  2000 

Visual  Displays 

Monochrome,  Stereo, 
1,000,000  pixels 

Color,  eye-limited 
resolution 

Audio  Displays 

User  Calib.,  Azimuth 
discrimination 

User  Indep.,  2  axis 
discrim 

Haptic  Displays 

Tactile  cueing,  force 
icons 

True  hand,  arm,  whole 
body  forces 

Head  &  Eye 
Transducers 

Head  -  1  arc  min,  eye  - 
5  degree 

Eye  -  30  arc  minute 

I 

Enabling  Technologies 


TECHNICAL  AREA 

BY  1995 

BY  2000 

Head-Mounted 

Displays 

Monochrome,  stereo, 
1,000,000  pixels 

Color,  eye-limited 
resolution 

Head-Mounted 

Earphone 

User  Calib.,  Azimuth 
discrimination 

User  indep. ,  2  axis 
discrim 

Force  Sensing  & 
Generation 

Tactile  cueing,  force 
icons 

True  hand,  arm,  whole 
body  forces 

Magnetic  & 
Optical 

Head-  1  arc  min,  eye  - 
5  degree 

Eye  -  30  arc  minute 
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DESCRIPTION  OF  TECHNOLOGY 

Most  military  tasks  involve  multiple  operators,  information  exchange  and 
transfer,  complex,  multi-component  decisions,  and  a  fast  work  tempo.  Such  a 
work  environment  requires  team  coordination  for  successful  completion  of 
tasks.  Team  training  teaches  the  skills  and  techniques  necessary  for  team 
members  to  accomplish  interdependent  tasks  as  a  coordinated  unit.  Team 
training  technology  provides  a  set  of  methodologies  and  instructional 
strategies  to  enhance  and  maintain  the  operational  proficiency  and  readiness 
of  combat  teams.  Four  support  systems  are  required  in  order  to  accomplish 
development  of  team  training  technology:  (a)  a  team  training  needs  analysis 
system,  (b)  a  team  performance  measurement  system,  (c)  a  team  training 
system,  and  (d)  an  instructor/observer  training  system. 

TEAM  TRAINING  NEEDS  ANALYSIS  SYSTEM 

Team  training  development  must  rest  on  a  comprehensive  understanding  of 
the  team  coordination  requirements  of  a  given  task.  The  needs  analysis 
survey  and  the  multiphase  analysis  of  performance  (MAP)  system  are  examples 
of  task  coordination  analysis  methodologies  that  are  being  developed  at  NTSC 
as  enabling  technologies  for  a  team  training  needs  analysis  system.  These 
methodologies  seek  to  identify  the  task-specific  teamwork  (i.e., 
coordination)  requirements  necessary  for  effective  team  performance,  and 
form  a  basis  for  training  development. 

Needs  Analysis  Survey.  Development  of  a  needs  analysis  survey  begins 
with  reviews  of  training  materials  and  team  research  programs.  This  process 
provides  background  information  about  behavioral  examples  of  teamwork 
skills.  In-depth  interviews  with  subject  matter  experts  in  the  training 
community  are  then  conducted  to  generate  behaviors  specific  to  that 
community  (e.g.,  aircrews  and  combat  information  center  operators).  A 
second  group  of  subject  matter  experts  verifies  the  importance,  difficulty, 
and  frequency  of  occurrence  of  these  behaviors.  Finally,  a  revised  survey 
instrument  is  given  to  a  larger  group  of  subject  matter  experts  to  establish 
that  the  interactive  skill  behaviors  are  critical  to  mission  effectiveness 
and  important  to  train  in  the  specific  community. 

Multiphase  Analysis  of  Performance  System.  The  MAP  system  is  a  process 
by  which  the  type  of  training  situation  drives  the  type  of  task  analysis 
method  to  be  used.  Four  unique  training  situations  dictate  which  team  task 
analysis  methodology  is  most  appropriate:  (a)  team-oriented  training  for 
inexperienced  teams,  (b)  team-oriented  training  for  experienced  teams,  (c) 
task-oriented  training  for  inexperienced  teams,  and  (d)  task-oriented 
training  for  experienced  teeuns.  Teeun  training  needs  and  objectives  are 
developed  from  the  task  analysis. 

TEAM  PERFORMANCE  MEASUREMENT  SYSTEM 

Viable  performance  measures  are  paramount  to  effective  training.  With 
respect  to  team  training,  measures  must  be  developed  that  allow  instructors 
to  diagnose  the  causes  of  effective  and  ineffective  team  performance.  These 
measures  also  form  the  basis  of  feedback  and  debrief  mechanisms.  In 
addition,  criterion  measures  are  necessary  so  that  training  effectiveness 
can  be  assessed  adequately.  Several  technologies  are  being  developed  to 
achieve  these  goals. 
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Measurement  of  Teamwork  Skills.  Critical  knowledge,  skills,  and 
abilities  (KSAs)  for  task-specific  team  coordination  are  generated  from  the 
needs  analysis  system.  Once  established,  KSAs  form  the  basis  for  measures 
of  teamwork  performance.  Specifically,  measures  of  team  process  can  be 
developed  by  constructing  observational  protocols,  and  by  employing  human 
performance  modeling  technologies  (e.g.,  petri  nets).  Standards  for  team 
performance  can  also  be  developed  using  scaling  techniques  (i.e.,  critical 
incidents) . 

Training  Evaluation.  Systematic,  objective  evaluation  of  team  training 
systems  is  required  to  assess  the  efficacy  of  training  interventions. 
Training  effectiveness  diagnostic  mechanisms  must  be  developed  in  order  to 
expedite  incorporation  of  "lessons  learned"  into  training  design.  Further, 
transfer  of  training,  that  is,  the  extent  to  which  training  produces  desired 
improvements  in  the  operational  environment,  must  also  be  assessed. 
Methodologies  that  provide  a  comprehensive  assessment  of  training 
effectiveness  must  be  developed. 

TEAM  TRAINING  SYSTEM 

Research  has  shown  that  teamwork  performance  is  critical  to  mission 
accomplishment.  The  goals  of  teaun  training  are  to  train  requisite  teamwork 
knowledge  and  critical  teamwork  skills,  foster  shared  task  models,  and 
maintain/enhance  training  motivation.  Achievement  of  these  goals  can  be 
accomplished  through  such  team  training  instructional  technologies  as: 

a.  Exercise  development  (e.g.,  lecture,  behavior  modeling,  role  play, 
simulation/exercises) , 

b.  Team  training  strategies  developed  for  specific  team  environments 
(e.g.,  crosstraining,  stress  exposure  training,  team  leader  training),  and 

c.  Computer  technology  (e.g.,  embedded  training,  computer  assisted 
instruction,  networked  systems). 

Instructional  strategies  are  designed  to  provide  trainees  with 
information  about  what  they  will  learn,  demonstration  of  expected  skills, 
opportunities  to  practice  skills,  and  feedback  on  training  performance. 

INSTRUCTOR/OBSERVER  TRAINING  SYSTEM 

The  training  of  instructors  and  observers  is  crucial  to  achieving  the 
teeun  training  objectives  listed  above.  Specific  instructional  technologies 
for  instructor/observer  training  include:  (a)  training  delivery  strategies 
that  include  effective  use  of  instructional  procedures  and  assessment 
instruments,  (b)  pre-training  capabilities  diagnosis,  and  (c)  task-related 
and  team-related  guided  feedback  training. 


CRITICAL  TECHNOLOGY  CHALLENGES  IN  TEAM  TRAINING 

Current  team  training  systems  lack  prescriptions  about  how  to  compose, 
manage,  and  train  teams.  NTSC  research  is  being  conducted  to  provide 
principles  and  guidelines  for  the  following  issues: 

Team  Training  Needs  Analysis  System 

Needs  Analysis  Survey 
Multiphase  Analysis  of  Performance 
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Team  Performance  Measurement  System 

Teiunwork  Skills  Measurement 
Training  Evaluation  Measures 
Diagnostic  Mechanisms 

Team  Training  System 

Exercise  Development 
Team  Training  Strategies 
Crosstraining 
Stress  Exposure  Training 
Team  Leader  Training 

Computer  Technology 
Embedded  Training 
Computer-Assisted  Instruction 
Networked  Systems  for  Team  Training 

Instructor/Observer  Training  System 
Training  Delivery 

Pre-Training  Capabilities  Diagnosis 
Task-Related  and  Team-Related  Guided  Feedback 


MILESTONES 


Team  Training  Needs  Analysis  System 


TECHNICAL  AREA 

BY  1995 

BY  2000 

Needs  Analysis  Survey 

Develop/Refine 

Guidelines  for 
Implementing 

Implement  in  new  or 
fielded  systems 

Multiphase  Analysis  of 
Performance 

Develop/Refine 

Guidelines  for 
Implementing 

Implement  in  new  or 
fielded  systems 
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MILESTONES  (CONTINUED) 

Team  Performance  Measurement  System 


TECHNICAL  AREA 

BY  1995 

BY  2000 

Teamwork  Skills 
Measurement 

Develop/Validate 
prototype  hviman 
performance  modeling 
measures 

Develop / Val idate 
prototype  critical 
incidents/scale 
development 

Provide  guidelines 
for  human  perfoirmance 
modeling  development 

Provide  guidelines 
for  performance 
measurement  development 

Training  Evaluation 
Measures 

Deve lop/ Val idate 
prototype  training 
evaluation  measures 

Provide  guidelines  for 
training  evaluation 
measures 

Diagnostic  Mechanisms 

Develop/Validate 
prototype  measures 

Provide  guidelines  for 
diagnostic  mechanism 
development 

Team  Training  System 


TECHNICAL  AREA 

BY  1995 

BY  2000 

Exercise  Development 

Develop/Refine 
principles  of  exercise 
development 

Provide  guidelines  for 
exercise  development 

Team  Training 

Strategies 

Develop/Refine 
principles  of 
crosstraining 
Develop/Refine 
principles  of  stress 
exposure  training 
development 

Identify  behavioral 
requirements  of 
effective  team 
leadership 

Provide  guidelines  for 

crosstraining 

development 

Provide  guidelines  for 
stress  exposure 
training  development 

Provide  guidelines  for 
tecun  leader  training 

Computer  Technology 

Establish  requirements 
for  embedded  team 
training 

Establish  requirements 
for  computer-assisted 
instruction 

Establish  requirements 
for  networked  team 
training  systems 

Provide  specifications 
for  embedded  team 
training 

Provide  specifications 
for  computer-assisted 
instruction 

Provide  specifications 
for  networked  tecun 
training  systems 
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Instructor/Observer  Training  System 


TECHNICAL  AREA 

BV  1995 

BY  2000 

Training  Delivery 

Establish  requirements 
of  training  delivery 

Develop  guidelines  for 
training  delivery 

Pre-Training 

Capabilities  Diagnosis 

Establish  requirements 
of  pre-training 
assessment 

Develop  guidelines  for 
pre-training  assessment 

Task-Related/Team- 
Related  Guided  Feedback 

Develop/Refine  feedback 
and  debrief  mechanisms 

Develop  guidelines  for 
feedback  and  debrief 
mechanisms 
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THREAT  MODELING  AND  COMMON  DATA  BASE 


DESCRIPTION  OF  TECHNOLOGY 

Simulation  of  the  tactical  environment  is  essential  for  meaningful 
mission  training  in  all  types  of  devices  used  for  training  in  tactical 
decision  making  and  weapons  systems  operations.  Training  must  be  provided 
for  missions  involving  electronic  warfare,  threat  recognition,  threat 
interception  and/or  avoidance,  terrain  masking,  countermeasures  use,  and 
offensive/defensive  weapons  deployment.  Elements  of  the  tactical 
environment  include  operational  platforms  (air,  surface,  ground,  etc), 
emitters,  IFF  responses,  active  and  passive  countermeasures,  weapon 
performance  and  lethality,  and  the  effects  of  ambient  conditions  on 
detectability  and  performance. 

Threat  models  have  typically  been  developed  on  a  custom  basis  to  suit 
individual  applications.  This  has  resulted  in  a  proliferation  of  models 
with  widely  varying  capabilities  in  terms  of  fidelity,  computational 
requirements,  growth  potential,  and  user/operator  interface.  Recent 
advances  in  computational  power,  simulator  interoperability,  and  visual 
image  generation  have  raised  user  expectations  for  threat  simulation  and 
emphasized  the  need  for  supplying  common  threat  models,  especially  for 
networked  training  devices.  Current  efforts  to  bring  some  order  to  the 
existing  chaotic  state  of  threat  modeling  have  revealed  the  following  key 
issues: 

Data  Sources.  The  data  required  to  describe  threat  weapon  system 
performance  and  threat  operator  behavior  are  gathered  by  multiple 
intelligence  agencies  for  a  multitude  of  purposes  not  related  to  real-time 
simulation  applications.  The  developers  of  threat  (and  "friendly”)  models 
must  gain  access  to  these  data  sources  and  then  overcome  data  shortcomings 
for  real-time  model  development. 

Common  Models.  The  diversity  of  threat  models  can  be  improved  to  reduce 
duplication  of  effort  by  focusing  on  user  requirements  and  associating  them 
with  model  complexity  and  fidelity.  Tactical  environment  simulations 
developed  for  battle  group  commander  training  lack  the  granularity  necessary 
to  support  individual  operator  training.  An  orderly  process  for  associating 
threat  model  capabilities  with  user  requirements  is  needed.  Performance 
metrics  must  be  developed  to  characterize  significant  implementation  issues 
such  as  computation  power  and  model  validation.  A  system  for  establishing 
and  maintaining  configuration  control  is  needed  for  model  credibility, 
commonality,  and  reusability. 

Human  Operator  Models.  Computer  programs  are  needed  to  control  the 
large  number  of  weapon  platforms  in  a  tactical  training  scenario.  This 
control  must  be  formulated  on  several  levels:  individual  platform  dynamics 
and  fire  control,  battery/site  operation,  and  overall  order  of  battle.  In 
many  applications,  the  user  desires  that  these  models  have  selectable  skill 
levels. 

Simulation  Architecture.  The  tactical  environment  simulation  must  be 
hosted  in  an  open  architecture  manner  to  ensure  maximum  growth  potential  and 
flexibility.  Architecture  design  must  not  preclude  the  use  of  emerging 
computer  capabilities  and  must  accommodate  a  wide  variety  of  model  types  and 
software  data  structures. 


C-35 


Special  Report  93-005 


User  Interface.  The  set-up,  control,  and  monitoring  of  tactical 
scenarios  must  be  optimized  so  that  individual  instructors  can  manage  these 
complex  problem  worlds  and  realize  effective  training.  Model  data 
structures  must  be  accessible  to  allow  modifications  by  users  for  "what  if" 
exercises,  automatic  scenario  generation  is  needed  to  simplify  instructor 
tasks,  and  meaningful  displays  and  parameters  must  be  defined  to  assist  in 
battle  management  and  debrief  roles. 

Interoperability/Correlation.  Tactical  environment  simulations  must  be 
capable  of  supporting  a  network  of  weapon  system  trainers,  including 
dissimilar  devices.  Threat  models  must  exhibit  common,  compatible 
characteristics  to  all  network  nodes.  Threat  databases  must  correlate  with 
all  other  databases  (visual,  radar,  IR,  etc.). 

MILESTONES 


TECHNICAL 

AREA 

BY  1995 

BY  2000 

Data  Sources 

Develop  process  to  coordinate 
access  to  data  sources  for 
weapon  system  performance  and 
operator  behavior 

Intelligence  agencies  fully 
integrated  into  threat 
modeling  process 

Common 

Models 

Develop  guidelines  and 
standards  for  user  needs, 
performance  metrics,  and 
configuration  control 

Establish  an  information 
clearing  house  for  reusable 
models 

Human 

Operator 

Models 

Develop  guidelines  for  model 
structure  and  selectable  skill 
levels 

Develop  standard  models  for 
several  classes  of  human 
model  applications 

User 

Interface 

Develop  guidelines  for 
automated  scenario  generation, 
model  access,  and  battle 
management  monitor  and  review 

Interoper¬ 

ability/ 

Correlation 

Demonstrate  network  of 
dissimilar  weapon  system 
trainers  with  common  threat 
environment 

Develop  guidelines  for 
network  operation 

Develop  fully  correlated 
data  sets 

C-36 


Special  Report  93-005 

AUTOMATIC  SCENARIO  GENERATION  AND  SCENARIO  CONTROL 


DESCRIPTION  OF  TECHNOLOGY 

New  concepts  are  required  for  effective  utilization  of  tactical  training 
systems  of  the  90s.  K  ten-fold  increase  in  the  total  number  of  tracks 
currently  simulated  for  tactical  training  systems  is  a  rec[uirement. 

However,  no  corresponding  increase  is  anticipated  in  the  number  of  training 
system  instructors  to  generate  or  control  training  system  scenarios  using 
this  increased  number  of  tracks. 

Scenario  generation  and  scenario  control  are  two  separate  and  distinct 
instructor  functions.  Scenario  generation  is  normally  an  off-line 
nonreal-time  function  of  initializing  forces  according  to  location/mission 
and  identifying  intended  movement  of  forces  according  to  a  timeline.  This 
function  is  required  even  in  those  systems  which  are  designed  primarily  as 
"non-canned"  or  "freeplay"  training  systems.  In  the  past,  scenario 
generation  or  the  development  of  one  tactical  scenario  could  take  as  long  as 
six  weeks  for  a  shore  based  tactical  training  facility  and  as  long  as  nine 
months  for  an  embedded  tactical  training  system. 

From  a  fleet  perspective,  the  issues  that  need  to  be  addressed  for 
scenario  generation  are:  (1)  ensure  the  economy  of  time  and  effort  for 
scenario  setup;  and  (2)  make  the  user-machine  interface  as  easy  as  possible 
to  use. 

The  training  system  instructor  scenario  control  function  consists  of 
both  monitoring  tracks  within  a  scenario  while  it  is  running  real-time  with 
students/team  and  also  providing  performance  measurement  and  feedback  of 
performance  in  a  timely  manner. 

Specific  fleet  requirements  for  1990b  scenario  control  are:  (1)  reduce 
the  instructor  workload,  e.g.,  assist  the  instructor  with  "modifying  track 
parameters"  such  as  course  and  speed  during  a  coordinated  attack  for 
multiple  tracks  in  a  multi-ship  training  evolution,  (2)  allow  instructor  to 
monitor  more  information,  and  (3)  provide  real-time  performance  measurement 
and  feedback  of  student  performance. 

Fleet  readiness  and  mission  effectiveness  will  be  enhanced  with  the 
application  of  new  technologies  to  automate  the  instructor  training  system 
functions  of  scenario  generation  and  scenario  control.  In  addition  to 
upgrading  existing  tactical  training  systems,  the  automation  of  these 
functions  will  be  critical  to  the  successful  implementation  of  new 
acquisitions  such  as  the  Tactical  Combat  Training  System  (TCTS). 

The  new  technologies  will  provide  rapid  development  and  operation  of 
training  system  exercises  that  are  representative  of  operational  events. 

The  time  required  to  create  a  typical  scenario  will  be  reduced  from  6  weeks 
to  1  week.  The  amount  of  information  required  to  specify  a  scenario  will  be 
reduced  by  over  90%.  During  control  of  scenarios,  instructors  will  be 
provided  multiple  windows  to  increase  the  amount  of  information  monitored, 
automatic  warfare  advisors  to  increase  instructor  response  to  rapidly 
changing  tactical  situations,  and  automatic  performance  measurement  and 
feedback  to  provide  timely  evaluations  of  exercise  successes. 

Major  technological  advances  in  expert  system  design,  hardware,  and 
software  are  now  available  to  automate  training  systems'  scenario  generation 
and  scenario  control  processes.  These  technological  advances  include  a 
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combination  of  one  or  more  of  the  following:  Expert  Systems/Neural  Nets; 
Data  Bases;  and  User  Interface  technologies. 

EXPERT  SYSTEMS/NEURAL  NETS 

Automatic  Initial  Force  Stationing.  Automatic  initial  force  stationing 
for  blue,  orange,  and  white  forces  is  technically  feasible  and  has  been 
demonstrated.  However,  the  most  efficient  design  and  grouping  of  rules  need 
to  be  assessed  and  documented. 

Track  and  Force  Modeling.  Current  scenario  generation/control 
capabilities  produce  scripted  scenarios.  These  scenarios  script  the  actions 
of  forces  so  that  without  trainer  intervention,  the  scripted  forces  follow  a 
time  tagged  series  of  events  after  scenario  start.  With  the  emphasis  now 
changing  from  scripted  scenarios  to  "free  play"  scenarios,  pseudo  forces 
need  to  be  designed  as  objects  that  take  actions  based  on  their  own  unique 
attributes,  capabilities,  missions,  and  a  set  of  rules.  The  rules  force  the 
object  to  react  to  actions  taken  by  other  objects. 

At  scenario  start  these  objects  take  actions  based  on  their  attributes, 
capabilities,  missions,  and  a  set  of  rules.  The  rules  force  the  object  to 
react  to  actions  taken  by  other  objects  (including  the  trainees)  within  the 
scenario  producing  a  dynamic,  intelligent  scenario. 

Automatic  Instructor  Situation  Assessment.  One  of  the  major  functions 
of  an  instructor  is  situation  assessment.  The  amount  of  information  an 
instructor  monitors  has  increased  and  will  continue  to  increase  in  the  90s. 
Automatic  situation  assessment  tools  would  reduce  the  instructor  workload 
while  allowing  him  to  monitor  more  information.  The  demonstration  of  an 
automatic  Anti-Air  Warfare  (AAW)  situation  assessment  tool  would  be 
developed. 

Automatic  Real-Time  Performance  Measurement  and  Feedback.  An  expert 
system  can  capture  the  rules  the  student  is  expected  to  follow  in  a  specific 
situation.  These  "built-in"  expert  system  rules  can  be  compared  to  the 
student's  actual  actions  in  real-time  and  provide  an  assessment  of 
student/group  performance.  A  prototype  expert  system  would  be  built. 

DATA  BASES 

Force  Element  Characteristics/Maps.  The  Navy  standard  data  bases  are 
currently  provided  in  a  relational  format.  Expert  system  shells  utilize  an 
object  oriented  data  base  format.  The  most  efficient  method  of  transforming 
this  relational  data  to  object  oriented  data  will  be  determined.  Navy 
standard  data  bases  are  large.  A  subset  of  the  data  base  may  be  sufficient 
for  automatic  scenario  generation  and  control.  Also,  the  choice  of  data  base 
to  support  the  display  of  maps  needs  to  be  determined. 

USER  INTERFACE 

Multi-Windowing  and  Graphics/Friendly  User  Machine  Interface. 
Multi-windowing  and  graphics  are  the  major  technologies  for  making  the 
user-machine  interface  easy  to  use.  Current  methods  for  instructors  to 
interface  with  training  systems  during  the  scenario  generation  process  is 
primarily  based  upon  "textual"  and  limited  graphics  presentations.  Recent 
studies  indicate  that  a  "picture"  or  "icon"  approach  can  improve  the  user's 
overall  productivity  by  75%.  The  incorporation  of  icons  into  the  scenario 
generation/control  process  can  significantly  improve  ease  of  use  and  reduce 
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operation  time.  The  issues  of  number  of  windows,  types  of  graphics,  and 
window  content  would  be  answered  through  prototyped  systems. 


MILESTONES 


TECHNICAL  AREA 

BY  1995 

BY  2000 

Expert 

o  Implement  Automatic  Force 

o  Implement  Automatic 

Systems/ 

Stationing  for  a  Battle  Group 

Force  Stationing  for 

Neural  Nets 

o  Provide  proof  of  concept 
demo  for  automatic  track  & 
force  movement  modeling  for 
multiple  ships 

o  Determine  requirements  for 
Automatic  Instructor 

Situation  Assessment  Aid 
o  Determine  requirements  for 
Automatic  Real-Time 

Performance  Measurement  and 
Feedback 

Battle  force 
o  Demo  automatic  track 
&  force  movement  for 
blue/orange  Battle 

Force 

o  Demo  Automatic 
Instructor  Situation 
Assessment  Aid 
o  Demo  Automatic  Real- 
Time  Performance 
Measurement  and 

Feedback 

Data  Bases 

o  Integrate  subset  of  Naval 
Warfare  Tactical  Data  Base 
into  Automatic  Scenario 
Generator 

o  Integrate  standard  Navy 
data  bases  for  maps  into 
Automatic  Scenario  Generator 

o  Integrate  entire 

NWTDB  into  ASG 

o  Re-evaluate  current 
standard  Navy  data 
bases  for  maps  and 
determine  if  new 
features  can  be  used 

User  Interface 

o  Demonstrate  system  that 
permits  user  to  modify  an 
Automatic  Scenario  Generator 
knowledge  base 
o  Demonstrate  use  of  multi¬ 
windowing,  graphics,  menus, 
to  enhance  user  productivity 

o  Complete  system  that 
permits  user  to  modify 

ASG  knowledge  base 

o  Complete 

implementation  of  user 
interface 
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DESCRIPTION  OF  TECHNOLOGY 

Artificial  intelligence  has  grown  rapidly  in  the  past  few  years  to 
encompass  a  wide  spectrum  of  technologies.  The  two  most  promising  for 
simulation  and  training  systems  technology  appear  to  be  neural  networks  and 
expert  systems.  Both  seek  to  transfer  knowledge  into  computers.  Expert 
systems  are  used  when  the  )uiowledge  can  be  translated  into  rules.  Neural 
networks  are  used  when  it  is  difficult  to  translate  the  knowledge  into 
rules,  but  it  is  easy  to  develop  examples  of  "good”  and  "bad”  performance. 
Generally,  it  is  so  expensive  to  develop  sufficient  examples,  that  neural 
network  technology  is  not  considered  ready  for  routine  application  in 
training  systems. 

Expert  systems  technology  does  appear  to  hold  near-term  promise  for 
training  systems  applications.  Its  impact  can  be  described  as  a  natural 
progression  of  simulation  technology,  itself.  In  one  sense,  many  training 
systems  use  a  range  of  techniques  to  create  the  illusion  of  reality.  For 
example,  mathematical  equations  of  motion  and  trajectory  generate  data  that 
drives  moving  targets  and  platforms  such  as  aircraft  and  ships.  The  FORTRAN 
language  was  a  great  improvement  over  assembly  language,  because  it  was 
easier  to  validate  the  mathematical  equations.  Models  are  used  to  present 
the  targets  and  platforms  in  a  sufficiently  realistic  form  that  the  student 
can  imagine  an  interaction  is  taking  place.  Expert  systems  technology  has 
the  potential  to  carry  knowledge  into  the  training  system  computer  that 
traditionally  was  carried  by  the  instructor  or  by  the  operator  acting  as  an 
adversary  to  the  student.  That  is,  both  knowledge  of  how  and  what  to  teach 
as  well  as  knowledge  of  human  interactions  can  be  represented  in  the 
training  system  using  expert  systems  technology.  Of  course,  clever 
programmers  have  had  the  capability  to  code  such  knowledge  in  FORTRAN  or 
other  popular  languages.  The  major  difference  with  expert  systems  is  that 
these  shells  allow  coding  of  knowledge  in  a  higher  order  language,  more  like 
experts  think.  Therefore,  it  is  easier  for  experts  to  validate  the  code. 
That  was  the  same  benefit  FORTRAN  gave  over  assembly  language  for 
programming  mathematical  equations.  Thus,  expert  systems  provides  the  next 
step  in  the  evolution  of  training  systems.  Three  applications  for  expert 
systems  that  have  had  recent  attention  are  intelligent  agents,  scenario 
control,  and  instructional  delivery. 

An  intelligent  agent  in  a  training  system  can  be  an  interactive  platform 
that  is  either  hostile  or  friendly.  The  goal  is  to  derive  from  the 
instructors  sufficient  knowledge  of  how  the  enemy  and  allied  forces  act  that 
agents  can  be  created  in  software  to  mimic  their  behavior.  Of  course,  the 
skill  must  also  vary  because  if  the  adversaries  are  too  good  and  nearly 
always  beat  the  student,  learning  may  not  take  place  efficiently. 

Therefore,  the  forces  must  adjust  to  the  fact  that  they  are  operating  in  a 
learning  environment.  Elements  include  tactics  and  rules  of  engagement. 

Scenario  control  in  a  training  system  can  serve  to  guide  both  the 
initial  force  laydown  in  a  military  training  operation  as  well  as  playout  of 
the  exercise.  Most  complex  command  and  control  training  systems  require 
many  days  and  sometimes  many  months  of  instructor  time  to  set  up  and  test 
any  one  scenario.  Unfortunately,  repeated  training  on  the  same  scenario  may 
result  in  students  learning  the  quirks  of  the  particular  scenario  instead  of 
the  concepts  of  engagement  intended  by  the  exercise.  Therefore,  an  expert 
system  which  would  develop  a  scenario  from  a  mission  objective  would  also 
give  practice  for  multiple  situations.  Elements  include  exercise 
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objectives,  strategy,  exercise  preview  and  playout,  short-term 
interventions,  and  review/replay. 

Instructional  delivery  in  a  training  system  can  provide  a  tutor  that 
tailors  the  course  to  the  individual  student.  This  will  ensure  that  each 
student  gains  the  maximum  knowledge  in  the  minimum  time.  It  might  also 
overcome  the  dilemma  of  having  the  best  pilot,  say,  being  the  worst 
instructor.  That  is,  an  expert  may  be  quite  good  at  doing  the  job,  but 
terrible  at  teaching  someone  else  how  to  do  it.  By  capturing  pedagogical 
knowledge,  a  training  system  can  teach  using  the  best  educational  practices. 
Of  course,  it  must  have  something  to  teach.  Thus,  instructional  delivery 
requires  domain  knowledge.  The  elements  of  pedagogical  knowledge  include 
assessment  (pre,  in  progress,  post),  presentation  (demonstrations,  guided 
practice,  free  play),  and  remediation  (revised  presentation,  determination 
of  asymptotic  performance  threshold).  The  elements  of  domain  knowledge 
include  the  material's  quality,  robustness,  reliability,  validity,  and 
comprehensiveness.  The  material  is  concepts,  facts,  examples,  analogies, 
and  performance  criteria. 

Certain  tools  are  needed  to  implement  expert  systems  technology  in 
training  systems.  Just  as  FORTRAN  required  flow  charts,  compilers, 
assemblers,  and  linking  libraries,  expert  system  technology  requires  tools 
and  techniques.  For  training  system  applications,  these  techniques  and 
tools  include  knowledge  acquisition,  knowledge  representation,  inference, 
test  case  performance,  documentation,  and  reusability. 

Knowledge  acquisition  is  akin  to  systems  analysis  in  traditional 
training  systems.  Many  attempts  have  been  made  at  developing  automated 
knowledge  acquisition  tools.  Many  books  are  written  about  it.  A  discipline 
has  developed  called  Knowledge  Engineering.  The  technique  is  still  in  its 
infancy.  The  dilemma  is  that  the  knowledge  of  the  expert  must  be  extracted 
and  translated  into  rules.  The  expert  often  does  not  articulate  the 
knowledge  in  a  way  that  the  programmer  can  translate  into  rules  easily. 
Likewise,  the  programmer  does  not  know  enough  about  the  domain  to  ask  the 
right  q[uestions  the  first  time.  Thus,  the  process  is  lengthy  and  iterative 
in  nature  for  complex  tasks.  There  are  tools  for  acquiring  knowledge  in 
specific  domains  for  specific  tasks.  However,  the  task  of  developing  a 
general  knowledge  acquisition  system  is  not  yet  complete.  The  elements  of 
knowledge  acquisition  include  requirements  analysis,  cost  of  allocation  of 
the  expert's  time,  "lost"  and  "unused"  knowledge  inventory,  and  revisions  to 
the  knowledge  base.  The  knowledge  that  must  be  acquired  is  addressed  in 
the  discussion  above. 

The  current  techniques  for  training  system  acquisition  include  task  and 
system  analysis.  The  task  and  system  analysis  of  a  training  system  has 
standard  reporting  formats.  Likewise,  knowledge  acquisition  from  a  domain 
expert  must  be  documented.  For  training  systems,  representation  of 
knowledge  is  critical  to  eventual  acceptance  testing.  Automated  acquisition 
systems  have  used  various  technic[ues  to  represent  what  is  acquired.  In 
general,  two  forms  of  knowledge  must  be  represented,  procedural  knowledge 
and  declarative  knowledge.  Declarative  knowledge  is  most  often  represented 
as  rules  and  procedural  knowledge  as  processes.  Elements  of  representation, 
as  applied  to  training  systems,  include  the  performance  specification, 
debugging,  validation  and  verification,  rule/process  revision,  and 
maintainability. 

Expert  systems  perform  their  work  through  a  process  of  inference.  As 
events  occur  in  a  training  system,  the  expert  system  must  check  through  its 
knowledge  base  of  rules  and  determine  whether  to  take  some  action.  Typical 
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inference  mechanisms  include  inductive  reasoning  (forward  chaining), 
deductive  reasoning  (backward  chaining),  and  probabilistic  reasoning. 
Essentially,  expert  systems  operate  on  elaborate  search  strategies, 
determining  which  "IF-THEN"  rules  apply  at  any  given  event.  The  search  can 
be  for  a  set  of  rules  that  match  the  conditions,  the  best  rule  that  matches, 
or  the  first  rule  that  matches. 

Acceptance  testing  of  a  training  device  validates  that  it  performs  as 
intended.  Generally,  test  cases  are  used  for  this  purpose.  For  training 
systems  that  incorporate  expert  systems  technology,  the  test  cases  must 
assure  that  the  system  "thinks"  like  the  expert.  All  important  variations 
must  be  tested  to  assure  the  system  will  perform  during  training.  One 
approach  to  development  of  test  cases  is  to  follow  psychological  guidelines 
on  human  sampling  of  skills.  Thus,  analysis  of  variance  techniques  can  be 
employed  to  test  the  system  the  way  one  might  test  a  human  subject.  This 
approach  appears  more  and  more  economical  as  the  knowledge  base  increases  in 
size. 

Most  training  systems  are  eventually  modified.  Documentation  of  rules 
of  the  expert  system  is  required  for  the  same  purpose  as  documentation  of 
the  rest  of  the  training  system,  to  assist  in  modification  and  maintenance. 
For  excunple,  the  domain  knowledge  may  change  as  additional  intelligence  is 
collected.  The  adversary  platforms  may  change  as  the  opposition  acquires 
advanced  technology.  Our  own  maneuvers  may  change  as  operational  forces 
learn  lessons  in  future  engagements.  Basis  of  documentation  for  expert 
systems  is  the  knowledge  representation  system.  It  must  map  into  rules.  In 
addition,  the  documentation  must  describe  the  interrelationship  of  the 
rules,  so  that  the  impact  of  modification  can  be  predicted. 

Reusability  of  software  is  gaining  popularity.  Certainly,  expert 
systems  can  incorporate  this  concept  in  its  knowledge  bases.  The  "IF-THEN" 
English-like  structure  of  rules  should  make  reusability  easier.  However, 
there  are  currently  no  standards  for  the  use  of  expert  systems  in  training 
systems.  As  applications  grow,  the  risk  of  unique  knowledge  bases  also 
grows.  The  elements  of  reusability  include  the  rules,  the  representation 
language  and  the  implementation  language. 

Explanation  is  a  human  process  that  is  often  used  to  validate  that 
communication  has  been  understood.  Most  expert  systems  use  an  explanation 
capability  as  a  debugging  tool.  The  system  can  explain  its  rule  sequence  to 
an  expert  when  the  expert  questions  the  conclusion  drawn  by  the  system.  The 
system  can  also  explain  its  rule  sequence  to  a  student  as  a  way  to  instill  a 
particular  logic  sequence  in  the  student's  performance  habits. 

CRITICAL  TECHNOLOGY  CHALLENGES  IN  EXPERT  SYSTEMS 

Intelligent  agents 
Scenario  Control 
Instructional 
Pedagogical 
Domain  Knowledge 
—  Knowledge  Acquisition 
—  Knowledge  Representation 
—  Inference 
—  Test  Case  Performance 

Documentation/Maintainability 
—  Reusability 
Explanation 
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TECHNICAL 

AREA 


Intelligent 

Agents 


Scenario 

Control 


Instruc¬ 

tional 

Pedagogical 

Domain 

Knowledge 


Knowledge 

Acquisition 


Knowledge 

Representa¬ 

tion 


Inference 


Test  Case 
Performance 


Documenta¬ 

tion 


Reusability 


Explanation 


BY  1995 


Establish  data  base  of 
various  platform  tactics, 
both  friendly  and  adversary 


Develop  automatic  scenario 
generation  workstation 
capabilities  to  reduce  time 
and  increase  flexibility 
Develop  automatic  scenario 
control  software  concepts  for 
friendly  and/or  opposing 
forces  to  use  intelligent 
agents  and  reduce  manning 


Develop  expert  teacher  for 
reuse  in  various  intelligent 
training  systems 

Develop  structure  for 
collection  of  domain 
knowledge  for  training 
systems 


Develop  automated  knowledge 
acquisition  system  for 
pedagogical  and  domain 
knowledge 


Develop  standards  for 
declarative  and  procedural 
knowledge  representation 


Develop  methods  of  test  case 
generation  for  pedagogical 
expert  systems 


Develop  techniques  for 
documentation  of  rules  and 
their  interrelationship 


Develop  techniques  for  reuse 
of  platforms  and  tactical 
rules 


Develop  techniques  for 
pedagogical  explanation  of 
domain  knowledge 


BY  2000 


Establish  repository  for 
high  quality  adversary  and 
friendly  platforms 


Develop  domain 
independent  expert  teacher 
for  curriculum  development 
and  control 

Develop  standards  for 
archiving  domain  knowledge 
of  the  "best"  operators 


Develop  domain  independent 
knowledge  acquisition 
system  for  theater 
waroamin 


Order  of  magnitude  speed 
increase  in  search 
strategies 


Develop  methods  of  test 
case  generation  for  domain 
expert  systems 


Develop  automatic 
debugging  tools  for 
knowledge  base 
modification 


Develop  standards  for 
expert  systems  shells 
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PAYOFF/VALUE  ADDED 


The  payoff/value  added  with  expert  systems  technology  accrues  to  the 
evolution  of  software  into  increasingly  English-like  code.  That  is,  expert 
system  technology  can  provide  higher  order  knowledge  beyond  that  of 
mathematical  equations  to  be  used  in  training  systems.  This  opens  the  door 
to  use  of  pedagogical  and  domain  knowledge  directly  instead  of  requiring 
software  analysts  and  engineers  to  translate  knowledge  into  lower  order 
computer  languages.  This  speeds  the  process  of  development,  debugging  and 
modification  of  training  systems  by  giving  development  tools  directly  to  the 
subject  matter  experts.  Further  benefits  accrue  when  the  training  systems 
become  richer  in  knowledge.  When  training  systems  have  resident  expert 
systems,  it  is  possible  to  reduced  manning  of  devices,  and  that  lowers  the 
cost  of  training.  At  the  same  time  the  system  continues  the  high  level  of 
knowledge  being  imparted  to  students  because  of  the  resident  expert  systems. 
Finally,  expert  systems  can  tailor  training  to  the  individual  student, 
thereby  promoting  maximum  learning  in  minimum  time. 
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DESCRIPTION  OF  TECHNOLOGY 

Visual/Sensor  simulation  is  a  technology  area  which  encompasses  a  number 
of  enabling  technologies.  The  enabling  technologies  are:  simulation 
requirements  analysis,  optics,  computer  image  generators/graphics,  light 
valves,  CRT's,  laser  displays,  liquid  crystal  displays,  digital  data  bases, 
photo  data  bases  visual  perception  and  variations  or  combinations  of  these. 

The  Visual/Sensor  simulation  areas  of  application  range  from  aircraft 
flight  simulators  with  fifty  foot  diameter  dome  displays  to  compact  helmet 
mounted  displays  or  night  vision  goggles  for  the  visual  areas.  In  the 
sensor  areas,  there  are  simulations  of  forward  looking  infrared  (FLIR)  Low 
Light  Level  (LLL)  TV  and  advanced  radars  such  as  synthetic  aperture  radar 
(SAR)  and  inverse  SAR  (ISAR).  Visual/Sensor  simulation  is  also  used  in  all 
types  of  weapon  fire  simulation  such  as  artillery,  tank  and  antitank  weapons 
and  infantry  weapons. 

Many  new  developments  in  the  enabling  technologies  are  reducing  cost, 
increasing  performance  and  decreasing  size  of  the  visual/sensor  simulation 
systems.  Computer  cost  reductions  and  capability  advances  lead  the  way  in 
computer  image  generation  technology.  New  developments  in  data  bases 
include  photographic  augmentation  of  CIG  systems  and  rapidly  reconf igurable 
data  bases  which  are  lowering  costs  and  increasing  realism  of  visual 
simulation  systems.  New  developments  in  light  valves  and  high  brightness 
CRT's  will  increase  brightness,  resolution,  and  reliability  of  projection 
systems. 

Developments  in  high  definition  TV  (HDTV)  should  lower  cost  and  improve 
performance  of  visual/sensor  simulation  systems  as  these  new  components 
become  available  from  mass  production  efforts.  The  new  components  will  be 
higher  resolution  TV  monitors  and  cameras,  low  cost  broad  bandwidth 
components,  and  network  capabilities  which  will  increase  the  use  of 
visual/sensor  simulation  in  networked  systems.  One  technology  push  that 
will  increase  performance  and  lower  cost  of  visual/sensor  technology  is  the 
development  of  Virtual  Reality  technology.  This  technology  will  put  in  the 
hands  of  the  public  the  ability  to  experience  or  live-out  their  virtual 
worlds  through  high  quality  visual  simulation  combined  with  other  sense 
simulations . 
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MILESTONES 


TECHNOLOGY  AREA 

BY  1995 

BY  2000 

High  Quality  Low 

Cost  CIG 

1990  Performance  at  1/10 
Cost 

PC  Size  and  Cost  with 
Performance  Limited  Only 
by  User's  Vision 

High  Quality  Light 
Weight  Optics  for 

HMD 

Etched  Plastic  Optics  Wide 
Angle  Well  Corrected 

Low  Cost  Helmet  Mounted 
Displays  (HMD's) 

Available  to  Public 

Advanced  CRT ' s  and 
Light  Valves 

2000  Lumen  CRT's  Low 
Maintenance  Light  Valves 

2000  Line  Resolution 

4000  Lumen  CRT ' s  Low 

Cost  Light  Valves 

Automated 

Photographic  Data 

Base  Generation 

Texture  Recognition  and 
Semi-Automated  Feature 
Extraction  from  Photos 

Neural  Networks  for 

Image  Analysis  and  Data 
Base  Generation 

HDTV 

Components  Designed  and 
Ready  for  Marketing 

In  General  Use  by  Public 
and  in  General  Use  in 
Networking  for  Training 
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APPENDIX  D 


SURVEY  DEVELOPERS  (D) ,  REVIEWERS  (V) ,  AND  RESPONDENTS  (R) 

AFFILIATION/ 

NAME  POSITION  TECHNOLOGY 

PAPERS 


o  Arthur  Banman  (V)  Chairman,  ADPA  Trng 

Systems  Division 


o  Robert  Breaux  (D,R)  NAVTRASYSCEN 

Psychologist 


AI/ Expert 
Systems 


o  Denis  Breglia  (D,R) 


NAVTRASYSCEN  Virtual  Env. 

Simulation  Imagery  Trng.  Technology 

Branch  Head 


o  Raymond  Brown  (R) 
o  Paul  Byrley  (V) 

o  J.  Cannon-Bowers  (D) 

o  Bob  Casullo  (V) 
o  Jim  Cooksey  (V) 
o  S.  H.  Cotton  (V) 
o  Ion  Deaton  (R) 

o  Glenn  Dillard  (R) 

o  G.G.  Dressel  (V) 
o  Bill  Findley  (R) 

o  Shirley  Flowers  (R) 


McDonnell  Douglas 
Trng  Sys  Staff  Director 

NAVTRASYSCEN 
Surface  Warfare  Applica¬ 
tions  Branch  Head 

NAVTRASYSCEN  Team  Training 

Psychologist 

Hughes  Training,  Inc. 

Pulau  Electronics 

Quintron  Corporation 

CAE-Link  Corp. 

Engineering  Manager 

NAVTRASYSCEN 

Engineer 

Quintron  Corporation 

NAVTRASYSCEN 

Engineer 

TRW/Trng  Sys  Engineering 
Section  Head 


o  Fred  Franz  (R) 


IntelliSys 

Manager,  Tactical  Appls 
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o  Thomas  Galloway  (D) 
o  Barry  Griffin  (R) 
o  Joan  Hall  (D) 
o  Robert  Hays  (D,R) 

o  Daniel  Herschler  (V) 

o  F.  Holland  (V) 
o  William  Holtsman  (R) 

o  Gerry  Kosydar  (R) 

o  Ed  Kulakowski  (V) 
o  Jeffrey  Marlin  (V) 

o  A1  Marshall  (R) 
o  Henry  Marshall  (R) 
o  Bruce  Montag  (R) 
o  Frank  Oharek  (D) 

o  Henry  Okraski  (V) 


NAVTRASYSCEN 

Engineer 

NAVTRASYSCEN 

Engineer 

NAVTRASYSCEN 

Psychologist 

NAVTRASYSCEN 
Training  Appls 
Branch  Head 


Threat  Modeling; 
Common  Data  Base 


Team  Training 


Embedded 

Trng. 


NAVTRASYSCEN 
Instructional  Sys 
Developer 

Hughes  Training,  Inc. 

CAE-LINK  Corp. 

Manager  of  IR&D 

CAE-Link  Corp. 

Dir,  Adv.  Eng.  &  Tech. 


Ref lectone 


Firearms  Trng  Sys,  Inc. 

Dir  of  New  Business 
Development 

NAVTRASYSCEN 

Engineer 

NAVTRASYSCEN 
Computer  Eng. 

Southwest  Research  Inst. 

Manager,  Sim  &  Modeling 

NAVTRASYSCEN  Visual/ Sensor 

Systems  Integration  Simulation 
Branch  Head 


NAVTRASYSCEN 
Research  and  Tech¬ 
nology  Director 
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o  William  Parrish  (D) 

o  Barbara  Pemberton  (D) 

o  William  Rizzo  (V) 

o  Joe  Rogers  (V) 

o  Eduardo  Salas  (D) 

o  Randy  Saunders  (R) 

o  Robert  Schwing  (V) 

o  G.  D.  Smith  (V) 
o  Terry  Snyder  (R) 

o  Richard  Soeldner  (R) 


NAVTRASYSCEN 
Systems  &  Tech 
Branch  Head 


NAVTRASYSCEN 

Engineer 


NAVTRASYSCEN 
Human,  Systems  Inte¬ 
gration  Division  Head 

NAVTRASYSCEN 
Systems  Engineering 
Division  Head 

NAVTRASYSCEN 

Psychologist 

Hughes  Training  Inc. 
Technical  Director 

Burtek 

Vice-President,  Ops 
Hughes  Training,  Inc. 
Grumman 

Project  Engineer, 
Sim/Trainer  Products 

NAVTRASYSCEN 
Systems  Development 
Branch  Head 
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Scenaroi  Gen  & 
Control 

Automated 

Scenario 

Gen  St  Control 


Team  Training 
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